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Preface KR. AWEDIKIAN 


PRÉFACE 


« Quand est ce qu’il faut arrêter de tester un produit logiciel ? » « Comment être sûr qu’un 
produit logiciel ne contient plus de défauts (bugs) et est prêt à être livré au client » et bien 
d’autres questions sur la qualité logicielle m’ont interpellé dès les premiers stages d’Ecole 
d'Ingénieur. En effet, diplômé de l’Ecole polytechnique de l’Université de Nantes en 2004, 
j'ai effectué 3 stages respectifs en 1°°, 2° et 3°" années. Durant ces stages, j’ai participé au 
développement de produits logiciels destinés à des applications PC mais aussi à des 
applications embarquées. À chaque fois qu’on développait un nouveau module logiciel, il 
nous fallait le tester. Le mot «tester » en industrie est souvent associé à tout type de 
techniques de vérification et de validation logicielle. Ayant appris en Ecole d’Ingénieur une 
panoplie de langages de développement informatique (C, C++, ...) et notamment comment 
concevoir et développer un produit logiciel, les tâches de développement informatique me 
paraissaient simples et maîftrisables. Mais, une fois le logiciel développé 1l faut le tester ; je 
me trouvais alors fort dépourvu méthodologiquement ! En effet, les formations actuelles 
d’Ingénieur logiciel se focalisent presque exclusivement sur le développement logiciel au 
détriment du test logiciel. Depuis les débuts du développement logiciel (Années 70), des 
chercheurs ont montré qu’il était illusoire de penser à effectuer un test logiciel exhaustif. Un 
Ingénieur doit toujours se contenter de tester un sous-ensemble de cas. Bien que certaines 
entreprises (les grandes) ont des processus bien définis pour tester un produit logiciel, la tâche 
de comment choisir les cas de test reste en grande partie basée sur l’expérience des 
Ingénieurs. Pour cela, et afin de tester les modules logiciels pendant mes stages, je choisissais 
certains cas de test en fonction de leur utilité et de leur efficacité mais aussi du temps qu’il me 
restait avant de devoir livrer le module. 


Après avoir obtenu le diplôme d’Ingénieur en 2004, je me suis intéressé plus généralement à 
la question : « Comment sont définis les processus de conception et conçus les méthodes et 
outils de conception de produits ». Afin de répondre à cette question, J'ai effectué un Master 
Recherche en ingénierie de conception au sein du laboratoire Génie Industriel de l’Ecole 
Centrale Paris. Le moment du stage arrivé, je me suis mis à la recherche d’un stage qui 
porterait sur l’amélioration des processus, méthodes et outils de conception de test logiciel 
pour établir une jonction entre mes domaines de prédilection. Bien heureusement, un stage sur 
le sujet était proposé par l’équipementier électronique automobile Johnson Controls. 
L’automobile, un secteur où l’électronique et le logiciel représentent plus de 30% du prix d’un 
véhicule. Pendant ce stage de 6 mois, nous avons mis en place une nouvelle approche de 
conception de cas de test logiciel. Le stage a livré des résultats prometteurs tant au niveau de 
la qualité du test logiciel que du temps passé pour tester un produit logiciel. De plus, nous 
avons pu identifier plusieurs pistes de recherche prometteuses. 


En se basant sur ces pistes de recherche, nous avons formulé un sujet de thèse de doctorat 
(avec Bernard Yannou' et Ludovic Augusto”) que nous avons proposé à la société Johnson 
Controls. En effet, il a fallu mettre en avance l’apport scientifique pour le laboratoire Génie 
Industriel et surtout l’apport industriel pour la société Johnson Controls qui a financé ce projet 
en partenariat avec l’ ANR sur un statut CIFRE. Suite à une réunion avec des responsables de 
la société et du laboratoire, l’accord pour lancer ce projet de thèse de doctorat a été donné en 
janvier 2006. Il est important de noter que la société Johnson Controls (France) n’avait jamais 
participé à un projet de thèse de doctorat auparavant. 


! Bernard Yannou, Professeur de l’Ecole Centrale Paris : encadrant de mon stage chez Johnson Controls 
? Ludovic Augusto, Chef de projet chez Johnson Controls : tuteur industriel de mon stage chez Johnson Controls 
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Résumé R. AWEDIKIAN 


RESUME 


L’électronique dans les voitures devient de plus en plus complexe et représente plus de 30% 
du coût global d’une voiture. Par exemple, dans une BMW série 5 modèle 2008, on peut 
trouver jusqu’à 80 calculateurs électroniques communiquant ensemble et représentant aux 
alentours de 10 millions de lignes de code logiciel. Face à cette montée en complexité, les 
constructeurs et équipementiers électroniques de l’automobile s’intéressent de plus en plus à 
des méthodes efficaces de développement, vérification et validation de modules électroniques. 
Plus précisément, ils focalisent leurs efforts sur la partie logicielle de ces modules puisqu'elle 
est à l’origine de plus de 80% des problèmes détectés sur ces produits. Dans ce contexte, nous 
avons mené un travail de recherche dont l’objectif est de proposer une approche globale 
d'amélioration de la qualité des logiciels embarqués dans les véhicules. Notre recherche part 
d’un audit des processus et outils actuellement utilisés dans l’industrie électronique 
automobile. Cet audit a permis d’identifier des leviers potentiels d'amélioration de la qualité 
logicielle. En se basant sur les résultats de l’audit et en tenant compte de la littérature dans le 
domaine de la qualité logicielle, nous avons proposé une approche globale de conception de 
cas de test pour les produits logiciels. En effet, nous avons développé une plateforme de 
génération automatique de tests pour un produit logiciel. Cette plateforme consiste à 
modéliser les spécifications du produit logiciel pour le simuler lors de tests, à se focaliser sur 
les tests critiques (ayant une forte probabilité de détecter des défauts) et à piloter la génération 
automatique des tests par des critères de qualité ; telles que la couverture du code et de la 
spécification mais aussi le coût des tests. La génération de tests critiques est rendue possible 
par la définition de profils d’utilisation réelle par produit logiciel, ainsi que par la réutilisation 
des défauts et des tests capitalisés sur des anciens projets. En plus des aspects algorithmiques 
du test logiciel, notre approche prend en compte des aspects organisationnels tels que la 
gestion des connaissances et des compétences et la gestion de projet logiciel. Notre approche 
a été mise en œuvre sur deux cas d’étude réels d’un équipementier électronique automobile, 
disposant de données de tests historiques. Les résultats de nos expérimentations révèlent des 
gains de qualité significatifs : plus de défauts sont trouvés plus tôt et en moins de temps. 


Mots clés: Vérification et validation logicielle, Automobile, Processus de test logiciel, 
Simulation fonctionnelle, Qualité logicielle, Gestion des connaissances, Prise de décision, 
Processus de conception. 


Quality of the design of test cases for automotive software: design platform and testing process 


9 


Abstract KR. AWEDIKIAN 


ABSTRACT 


Nowadays, car electronics become more and more complex and represents more than 30% of 
the total cost of a car. For instance, in a 2008 BMW 5 series model, one can find up to 80 
electronic modules communicating together and representing 10 million lines of software 
code. Facing this growing complexity, carmakers and automotive electronic suppliers are 
looking for efficient methods to develop, verify and validate electronic modules. In fact, they 
focus on the software part of these modules since it accounts for more than 80% of the total 
number of problems detected on these modules. In this context, we achieved our research 
project with the aim of proposing a global approach able to improve the quality of automotive 
embedded software. We started with an audit of the software practices currently used in 
automotive industry and we pinpointed potential levers to improve the global software 
quality. Based on the results of the audit and the literature review related to software quality, 
we developed a global approach to improve the design of test cases for software products. In 
fact, we developed a test generation platform to automatically generate test cases for a 
software product. It is mainly based on modeling the software functional requirements in 
order to be simulated when testing the software, focusing on critical tests to be done (because 
of their higher probability to detect a bug) and monitoring the automatic generation of tests by 
quality indicators such as the structural and functional coverage but also the tests cost. The 
generation of critical tests is based on the definition of real use profiles by software product 
and on the reuse of bugs and test cases capitalized on previous projects. Besides the 
computational aspects of software testing, our approach takes into account organizational 
matters such as knowledge management, competency management and project management. 
Our approach have been implemented in a computer platform and experimented on two 
typical case studies of an automotive electronic supplier, with historical test data. The results 
of our experiments reveal significant improvement in software quality: more bugs are 
detected earlier and in less time. 


Keywords: Software verification and validation, Automotive, Software testing process, 
Functional simulation, Software quality, Knowledge management, Decision making, Design 
process. 
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I. Context 


Nowadays, electronics represents more than 30% of the global cost of a car. Car electronic 
architecture becomes more and more complex and carmakers outsource the design of 
electronic modules to automotive electronic suppliers. The software part is the added value of 
these modules and they account for more than 80% of the total number of defects detected on 
these modules. As automotive electronic products become more and more complex, the size 
of software embedded in these products increases drastically. As a consequence, the time 
spent in verifying and validating these software has increased exponentially the last 10 years. 
Verification and Validation (V&V) activities account now for more than 50% of an 
automotive electronic project time and effort. Despite the huge resources spent in verifying 
and validating a software product and after each delivery to the carmaker (up to 10 may be 
made), some bugs are still detected by the carmaker and forwarded to the supplier who must 
react quickly and efficiently. Once an electronic module is launched on the market (e.g. 
integrated into a vehicle), an average of one software bug per year is detected by the end- 
users, which may becomes dramatic for the electronic supplier in financial and image terms 1f 
the product has to be systematically substituted. 


As the automotive market becomes more and more competing, decreasing the development 
me of outsourced parts and lowering the number of defects detected later in the process 
becomes of major importance for carmakers and, consequently, a major quality indicator for 
automotive suppliers. Indeed, the carmakers” process for assigning new projects to suppliers 
is mainly based on feedbacks from previous projects. Consequently, suppliers work on 
reducing the development time of their products, delivering on time the products to carmakers 
and detecting the maximum number of bugs as early as possible in the development process. 


Through our research project (PhD), we were asked by Johnson Controls, one of the world’s 
leading suppliers of automotive interior systems, electronics and batteries, to improve the 
performance of its software V&V activities. Their main purpose is to improve the quality of 
their products and therefore better satisfy the requirements and expectations of their clients. In 
our research (Awedikian 2007), we go through this problem with a systemic approach in 
order to identify levers in any domains from which we might be able to improve the global 
performance of the software V&V activities. The major added value of the present work 1s to 
globally solve the quality issue of the software testing process. 


IL. Research process 


Our research process is based on five main stages: 
Stage 1: Industrial audit 


The audit of the industrial context aims to identify and determine the overall 
environment of our research project. This results in identifying a list of anomalies and 
issues in the current verification and validation practices. 


Stage 2: Research topic definition 


Based on the results of the industrial audit, the definition of our research topic allows 
to better determine the scope and focus of our research. This also leads to a better 
definition of the state-of-the-art focus. 


Stage 3: State-of-the-art 
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The state-of-the-art on the research issues in the scope of our research pinpoints 
existing solutions; their advantages, drawbacks and adaptability to our context. This 
results in a list of potential proposals. 


Stage 4: New concepts development 


The development of new concepts 1s the result of the three previous stages. Based on 
the concepts identified in the literature and taking the requirements of our industrial 
context into account, new concepts are developed. 


Stage 5: Concepts prototyping and validation 


The prototype development aims to implement our new concepts in a computer 
platform. This platform gives us the opportunity to validate our concepts on typical 
case studies. 


Figure Introduction.1 illustrates our research stages all along the three years of the PhD 
CUrSUS. 


Stage 


Industrial 
audit 


Research topic 
definition 


Stage 3 
State-of- 
the-art 


Stage 4 
New concepts 


develop ment 


Stage 5 


Concepts prototyping and validation 


1“ year 2"d year 3'dyear Time 
Figure Introduction.1 — Stages of our research process 


IIT. Contributions’ overview 


Through our research project, we perform an audit on the current software practices in 
automotive industry. The result of the audit is a list of anomalies and lacks (diagnoses) in the 
current software V&V activities in automotive industry. Based on the audit results and the 
literature review, we propose a new systemic approach to automate éfficiently the design of 


Quality of the design of test cases for automotive software: design platform and testing process 


34 


General introduction R. AWEDIKIAN 


test cases for software products. Our approach presents a much different workflow than the 
one presently used in automotive industry. The new workflow is based on eight activities 
which are manual, semi-automatic or automatic and managed by different individuals. These 
activities are: 


1. Model the software functional requirements using a formal simulation model that we 
developed keeping in mind the automotive context and its constraints (Awedikian 
2008a). 

2. Verify and validate the consistency and compliance of the requirements model 
(Awedikian 2008àa). 

3. Define some behavioral characteristics of a car driver when using the software product 
under test (Awedikian 2008b). 

4. Reuse the test cases developed in the past for similar software products (Awedikian 
2008b). 

5. Reuse the bugs detected in the past on similar software products (Awedikian 2008b). 

6. Enrich the requirements model with knowledge (from activity 3 to 5) on the driver 
recurrent operations and the test engineers’ experience (Awedikian 2008b). 

7. Automate the design of fest cases from the enriched model of functional requirements 
(Awedikian 2008c). 

8. Monitor the design of test cases by quality objectives but also time and cost 
constraints (Awedikian 2008c). 


Processes, roles and tools implementing these activities have been developed. The results of 
the experiment of our approach on two typical industrial case studies (within Johnson 
Controls) are very promising. We reduce by 70% the number of bugs detected by the 
carmakers and by 9% the ones detected by the end-users. Moreover, we reduce by 22% the 
üme spent in testing a software product. In fact, we detect the bugs earlier in the software 
development process and closer to their origin. We also propose to deliver to the carmaker 
Jormal quality indicators on the delivered software. All these results contribute to an 
improvement of the customer satisfaction and as a direct impact; the number of tenders will 
grow. Unfortunately, estimating the cost of software bugs in an organization is a delicate, 
strategic and confidential question and therefore we were not allowed to communicate the 
numbers on the bugs’ costs savings via the use of our approach. 


As a consequence of these results, managers at Johnson Controls decide to patent our 
approach’. However and in order to patent an idea in Johnson Controls, a formal verification 
and validation process of the idea is required. In our case and before starting this process, 
Johnson Controls has submitted a worldwide Quick Patent” in order to protect our approach. 
A worldwide survey on software testing approaches has been performed by Johnson Controls 
patent experts. Moreover, we were formally interviewed by many managers and experts on 
the contributions of our approach. The final stage will be the decision of the Johnson Controls 
Intellectual Property (IP) committee to patent or not our approach. 


IV. Reading guidelines 


This dissertation is composed of 4 parts and each part is composed of two or more chapters. 
The structure of the document is illustrated in Figure Introduction.2: 


* Including Bernard Yannou and Mounib Mekhilef as co-inventors. 
* In France, we associate a Quick Patent to an “Enveloppe Soleau” (http://www.inpi.fr/fr/services-et- 
prestations/enveloppe-soleau.html, Consulted on November 2008). 
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e Part I develops the research context and the industrial audit (Chapters 1 and 2). 

e Part II develops the research topic and the literature review (Chapters 3 and 4). 

e Part III develops our new concepts (Chapters 5, 6, 7 and 8). 

e _ Part IV develops the computer implementation and the validation of our new concepts 
(Chapters 9 and 10). 

General Part I - Part II - Part III - Part IV - General 
introduction | Context and domain Problem statement A new approach for Implementation, conclusion 
analysis and literature designing efficient test validation and impacts of 

review cases for a software product the proposed approach 
Chapter 1. Chapter 3. Chapter 5. Chapter 9. 
The need to Research topic Modeling and simulation Prototype 
improve the of software functional implementation 
quality of Chapter 4. requirements 
software products State-of-the-art Chapter 10. 
in automotive Chapter 6. Modeling and 
industry Verification and simulating two 
validation of a software industrial case studies 
Chapter 2. functional requirements 
Industrial audit model 
Chapter 7. 
Automatic generation of 
test cases 
Chapter 8. 


Refining the operation 
space description with 
the driver behavior's 
profile, past bugs and 
test cases 


Figure Introduction.2 - Document structure 
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CHAPTER 1. THE NEED TO IMPROVE THE 
QUALITY OF SOFTWARE PRODUCTS IN 
AUTOMOTIVE INDUSTRY 
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I. Introduction 


The world of electronics 1s living a revolution in the way products are imagined, designed and 
implemented. The ever growing importance of the internet, the advent of microprocessors of 
great computational power, the burden of wireless communication, the development of new 
generations of integrated sensors and actuators are changing the world we live and work in. 


The car as a self-contained universe is experiencing a similar revolution. We need to rethink 
what a “car” really is and the role of embedded electronics. Electronics 1s now essential to 
control the movements of a car, of the chemical and electrical processes taking place inside, to 
entertain the passengers, to constantly be connected with the rest of the world and to ensure 
safety. However, the growth of electronics in a car might reduce its reliability. Will 
electronics take the major role in car manufacturing and design? How to control the quality 
of electronic systems? How to manage the growth of software complexity in automotive 
electronic parts? What will an automobile manufacturer’s core competence be in the next few 
years? What are the new challenges for automotive electronic suppliers ? 


Our intent in this chapter 1s to answer some of these questions. An illustration of the present 
automotive industry context facing the globalization and the outsourcing issues is carried out 
in Section 2. Then, the electronic architecture of a modern vehicle is described in Section 3, 
showing the tendency toward incorporating ever more electronics. An overview of the role of 
software and the new challenges in automotive electronics industry is done in Section 4 and 5. 
Finally, the industrial needs and expectations as it was expressed for the first time by Johnson 
Controls company are summarized in Section 6. 


IL. The phenomena of globalization and outsourcing in automotive 
industry 


As we enter the new millennium, globalization has emerged as one of the most salient and 
powerful forces shaping domestic and world economies. 


Definition 1.1: Globalization (Wikipedia -— November 2008) 


Globalization in its literal sense is the process of transformation of local or regional things or 
phenomena into global ones. It can also be used to describe a process by which the people of 
the world are unified into a single society and function together. This process is a 
combination of economic, technological, sociocultural and political forces. Globalization is 
often used to refer to economic globalization, that is, integration of national economies into 
the international economy through trade, foreign direct investment, capital flows, migration, 
and the spread of technology. 


The automobile industry is typically considered to be at the forefront of globalization. 
Evidences supporting this view have been listed by Spatz (Spatz 2002): 


e the intricate network of alliances and cross-shareholdings among automobile 
companies, Within nations and regions but also between regions, 

e intensified Mergers and Acquisitions (M&A) activities in the 1990s, involving both 
end-producers and automotive input suppliers, 

° and the trend towards technologically motivated cooperation agreements, which was 
caused, inter alia, by end-producers entering into new forms of partnerships for the 
design of principal modules and subsystems. 
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The new face of globalization in automotive industry 1s best revealed by the rise of global 
suppliers. Companies such as Johnson Controls, Bosch, Denso, Lear Corporation, TRW, 
Magna, and Valeo have become the preferred suppliers for automakers around the world. 
Some automakers, particularly American firms, have combined a move to “modular” final 
assembly with increased outsourcing, giving increased responsibility to first-tier suppliers” for 
module design and second-tier sourcing. Many first tier-suppliers started to build a vertical 
integration (through mergers, acquisitions, and joint-ventures) and to geographically spread 
so as to be able to provide their customers with modules on a worldwide basis. At the same 
me, it can be simultaneously observed a deverticalization in automaker companies which 
leads to create a new global-scale supply-base capable of supporting the activities of final 
assemblers on a worldwide basis. 


The drivers of increased outsourcing include 1) the rising technological complexity of vehicle 
development, 2) rising logistics complexity as more production locations come on-stream, 3) 
a desire to “streamline” the final assembly process, 4) a desire to pay for parts only as they are 
Incorporated into vehicles rather than when they are shipped from suppliers, 5) increasing 
competence in suppliers, and 6) a desire to lower costs by moving production to low cost 
suppliers. 


Twenty years ago, automakers practiced low-level parts assembly within final assembly 
plants, purchased parts based on price, and paid minimal attention to quality. Now, 
automakers ask suppliers to do more design and sub-assembly work. This refers to as 
“modularization” in the automotive industry. For example, vehicle doors can be delivered 
with the glass, fabric, interior panels, handles, and mirrors pre-assembled. Dashboards can be 
delivered complete with polymers, wood, displays, lights, and switches. The aim of 
modularization 1s to move labor out of the final assembly process (design for 
manufacturability can serve the same purpose). 


According to Sturgeon (Sturgeon 2000), fifteen modules represent about 75% of vehicle 
value. In fact, a supplier can provide groups of related modules, called “module systems”. For 
example, seats, interior trim, and cockpit modules could be supplied as a complete “interior 
system”. Figure 1.1 provides a graphic representation of the apparent trend from discrete parts 
to modules and systems. 


$ First-tier supplier means a supplier who directly provides goods and services to the assembly plant of the 
product 
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Figure 1.1 - From part to module to system (Sturgeon 2000) 


Strong growth forecast for electronic parts in automotive 


The past four decades have known an exponential increase in the number and sophistication 
of electronic systems in vehicles. According to Leen (Leen 2002), nowadays, the cost of 
electronics in luxury vehicles may amount to more than 23 percent of the total manufacturing 
cost. According to Moavenzadeh (Moavenzadeh 2006), a top R&D° executive from General 
Motors said that electronics and software content will account for 40% of the value-added in 
the vehicle over the new ten years. Moreover, a recent quote” from Daimler executives says 
that more than 80% of innovation in the automotive domain will be in electronic modules. 


The growth of electronic systems has had implications for vehicle engineering. For example, 
today’s vehicles may have more than 4 kilometers of wiring, compared to 45 meters in 
vehicles manufactured in 1955. In July 1969, Apollo 11 employed a little more than 150 
Kbytes of onboard memory to go to the moon and back. Just 30 years later, a family car might 
use 500 Kbytes to keep the CD player from skipping tracks. 


The resulting demands on power and design have led to innovations in electronic networks for 
automobiles. Researchers have focused on developing electronic systems that safely and 
efficiently replace entire mechanical and hydraulic applications. Just as LANS* connect 
computers, control networks connect a vehicle’s electronic equipment. These networks 
facilitate the sharing of information and resources among the distributed applications. In the 
past, wiring was the standard means of connecting one element to another. As electronic 


R&D: Research and Development 
7http://www.dsp.acm.org/view_lecture.cfm?lecture_id=86 (consulted on November 2008) 
$ LAN: Local Area Network 
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content increased, the use of more and more discrete wiring hit a technological wall. 
Fortunately, today’s control and communications networks, based on serial protocols, counter 
the problems of large amounts of discrete wiring. Beginning in the early 1980s, centralized 
and then distributed networks have replaced point-to-point wiring. 


Figure 1.2 shows the sheer number of systems and applications contained in a modern 
automobile’s network architecture. 
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Figure 1.2 —- A modern vehicle’s network architecture (Leen 2002) 


The electronics and software content of vehicles relies more on electrical and software 
engineers than on the traditional mechanical engineers associated with the automotive 
industry. Moavenzadeh (Moavenzadeh 2006) indicates that electronics and software 
engineering functions are easier to outsource or offshore than mechanical engineering 
functions. Software engineers across an ocean can discuss a few lines of code easier than 
mechanical engineers can discuss how to modify the design of a module. Software and 
electronic systems also tend to follow a more modular product architecture than mechanical 
systems; therefore, it 1s easier to offshore both low and high value added functions. 


IV. Challenges for the automotive electronic suppliers 


The overall goal of electronic embedded system design is to balance production costs with 
development time and cost in view of performance and functionality considerations. In other 
words, engineers are encouraged to shorten the overall design and validation cycle without 
compromising quality, reliability, and cost targets. 
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A. Lower the production cost 


Manufacturing costs mainly depend on the hardware modules of the product. If one considers 
an integrated circuit implementation, the size of the chip 1s an important factor in determining 
production cost. Minimizing the size of the chip implies tailoring the hardware architecture to 
the functionality of the product. However, the cost of a state-of-the-art fabrication facility 
continues to grow up. In addition, the Non-Recurring Engineering (NRE) costs associated 
with the design and tooling of complex chips are rapidly growing. As a consequence, a 
common hardware platform to be shared across multiple applications may increase the 
production volume and decrease the overall cost. 


B. Lower the development time and cost 


Since the times of ignition car electronics in the 1970s, the complexity of automotive 
electronics architecture is still growing. Presently, Leen (Leen 2002) notices that a BMW 5 
series model can have up to 80 electronic control units. However, the market dynamics for 
automotive electronic systems leads to shorter and shorter development times. Presently, no 
matter how complex the design problem is, suppliers don’t have more than six months from 
the delivery of the customer requirements to a first and correct implementation. To meet the 
design time requirements and ensure a high quality of the delivered product, a design 
methodology that favors automation, reuse and early problem detection is essential. This 
implies that the design activity must be rigorously defined, so that all stages are clearly 
identified and appropriate checks are enforced. 


V. The role of ‘‘software” in automotive electronics 


Lets us start by defining what is a software. In this dissertation, we adopt the definition of 
software proposed by the Institute of Electrical and Electronics Engineers” (IEEE). 


Definition 1.2: Software or Software product (IEEE Std. 610-1990) - Abbreviation: SW 


Software is a general term used to describe a collection of computer programs, procedures, 
and possibly associated documentation and data pertaining to the operation of a computer 
system. 


Moreover, a software product is composed from a set of software components or modules (Cf. 
Figure 1.3). 


Software product 


Inputs 


Outputs 


Figure 1.3 - Software product versus software component 


* http://www.ieee.org/portal/site (Consulted on November 2008) 


Quality of the design of test cases for automotive software: design platform and testing process 


45 


The need to improve the quality of software products in automotive industry R. AWEDIKIAN 


Definition 1.3: Software component or module (Wikipedia - November 2008) 


A software component or a software module is a system element offering a predefined service 
or event, and able to communicate with other components or modules. It is a minimal 
software item that can be tested in isolation. 


A software product can be dedicated for different types of electronics architecture (computer, 
automotive, airplane etc.). In our research, we only consider the software products written for 
machines that are not, first and foremost, computers. In software engineering, this type of 
software products 1s called embedded software. For example, it is embedded to electronics in 
cars, telephones, audio equipment, robots, appliances, toys, security systems, pacemakers, 
televisions and digital watches. This type of software products can become very sophisticated 
in applications like airplanes, missiles, process control systems, and so on. Embedded 
software 1s usually written for special purpose electronics architecture. For instance, the 
CPUs'° are different from general purpose CPUs that we could find in our desktops or 
laptops. Moreover, a real-time operating system is required for managing the simulation of 
embedded software. In fact, tasks’ scheduler and priorities are the fundamentals of a real-time 
operating system. 


The design process of software products shares many technologies with the one of hardware 
products. Nevertheless, there are important differences between the two types. In the 
following, we identify some of these differences: 


e the hardware product quality relies on design, implementation and manufacturing 
processes, however the software product quality relies only on design and 
implementation processes. The software manufacturing process is mainly based on a 
“simple” reproduction activity, 

° _a software product is able to simulate alternative commandés on different inputs which 
lead to a high complexity of the product, 

e _a software product is not a physical entity and therefore, it doesn’t wear out over time. 
In fact, since problems are detected and corrected, the quality of a software product 
improves over time. However, the correction and/or evolution activities can introduce 
new problems in the product, 

e software problems cannot be prevented. In fact, specific successive commands on the 
inputs of the software product can reveal a problem which was not detected during the 
testing activities of the product, 

e the easiness and the rapidity which with a software product can be modified lead to the 
fact the software development process should be very well monitored and 
documented, 

° and historically, software modules are not frequently standardized and reused. 
Nowadays, there is a trend toward a reuse of software modules in order to lower the 
development time and cost. 


One more specific concept to software products is the size. Software sizing (Wikipedia — 
November 2008) is an important activity in software engineering that is used to estimate the 
size of a software module. Size 1s an inherent characteristic of a software in just like weight 1s 
an inherent characteristic of any tangible material. Historically, the most common software 
sizing methodology was counting the Lines Of Code (LOC) written in the application source. 
Another famous sizing method is the Function Point analysis. New trends of software sizing 
have recently emerged. 


"© CPU: Central Processing Unit 
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A. Growth of software size in automotive electronic parts 


The amount of software in many electronic products 1s increasing rapidly. For example, the 
number of lines of source code in a mobile phone is expected to increase from 2 million today 
to 20 million by 2010; à car will contain 100 million lines of code (Charrette 2005). 
Consequently, electronics companies no longer find it economically viable to provide all the 
software in their products. 


In fact, the amount of software in cars grows exponentially. Within only thirty years, the 
amount of software in a car has evolved from zero to tens of millions lines of code. A current 
premium car, for instance, implements about 270 functions a user interacts with, deployed 
over about 70 embedded platforms. Altogether, the software amounts to about 100 megabytes 
of binary code. The next generation of upper class vehicles, hitting the market in about 5 
years, is expected to run up to 1 gigabyte of software. This is comparable to what a typical 
desktop workstation runs today. 


A first reason for this growing complexity in software is that software enables the 
implementation of functionality deemed impossible just twenty years ago. Another reason is 
that electronics in cars helps to reduce gas consumption and increase performance, comfort 
and safety, as indicated by the decreasing number of major accidents whereas traffic 
increases. Information processing technology cuts across all aspects of the car and is a 
persuasive, sophisticated and differentiating value addition to the product. Furthermore, 
software enables the car manufacturers and suppliers to tailor systems to particular customers’ 
needs. In other words, software can help differentiate between cars. At least in principle, it is 
the software that also allows hardware to be reused across different cars. Contrarily to 
hardware, software has an almost negligible replication cost, which is a further incentive to 
bet on software as a potential tool in cost-reduction. However, the growing complexity of 
automotive software products leads to a dramatic increase of the software development costs. 
In addition, growing complexity is a driver for numerous challenges in the automotive 
industries like: definition of key competencies, processes, methods, tools, models, product 
structures, division of labor, logistics, maintenance, and long term strategies. 


B. Software Development Life Cycle 


The Software Development Life Cycle (SDLC) models describe activities of the software cycle 
and the order in which those activities are executed. A variety of SDLC models have been 
proposed in a paper (Green 1998), most of which focus exclusively on the development 
activities: ad-hoc model, waterfall model, V-model, iterative and incremental model, 
prototyping model, rapid application development model, exploratory model and spiral 
model. 


1. Ad-hoc model 


Early systems development often took place in a rather chaotic and haphazard manner, relying 
entirely on the skills and experience of the individual staff members performing the work (Cf. 
Figure 1.4). Today, many organizations still practice Ad-hoc Development either entirely or 
for a certain subset of their development (e.g. small projects). 
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Figure 1.4 - Development based on the skills and experience of the individual staff 
members performing the work 


In the absence of an organization-wide software process, repeating results depends entirely on 
having the same individuals available for the next project. Success that rests solely on the 
availability of specific individuals provides no basis for long-term productivity and quality 
improvement throughout an organization. 


2. The waterfall model 


The waterfall model prescribes a sequential execution of a set of development and 
management activities (Cf. Figure 1.5). Some variants of the waterfall model allow revisiting 
the immediately preceding activity ("feedback loops") if inconsistencies or new problems are 
encountered during the current activity. The waterfall model is the earliest method of 
structured system development. Although it has been criticized in recent years for being too 
rigid and unrealistic when it comes to quickly meeting customer’s needs, the waterfall model 
1s still widely used. It provides the theoretical basis for other process models. 
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3. The V-model 


A variant of the waterfall model - the V-model - associates each development activity with a 
Verification and Validation (V&V) activity at the same level of abstraction (Cf. Figure 1.6). 
Each development activity builds a more detailed model of the system than the one before it, 
and each V&V activity tests a higher abstraction than its predecessor. 
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Figure 1.6 — V development model (V-model) 
4, The iterative and incremental model 


The problems with the waterfall model created a demand for a new method of developing 
systems which could provide faster results, require less up-front information, and offer greater 
flexibility (Cf. Figure 1.7). With iterative development, the project is divided into small parts. 
This allows the development team to demonstrate results earlier in the process and obtain 
valuable feedback from system users. Often, each iteration 1s actually a mini-waterfall process 
with the feedback from one activity providing vital information for the design of the next 
activity. 
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Figure 1.7 — Iterative and incremental development model 
LÉ The prototyping model 


The prototyping model was developed on the assumption that it is often difficult to know all 
of your requirements at the beginning of a project. Typically, users know many of the 
objectives that they wish to address with a system, but they do not know all the nuances of the 
data, nor do they know the details of the system functionalities and capabilities. The 
prototyping model allows for these conditions, and offers a development approach that yields 
results without first requiring all information up-front. 


When using the prototyping model, the developer builds a simplified version of the proposed 
system and presents it to the customer for consideration as part of the development process. 
The customer in turn provides feedback to the developer, who goes back to refine the system 
requirements to incorporate the additional information. 
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6. The rapid application development model 


A popular variation of the prototyping model is called Rapid Application Development (RAD). 
RAD focuses on developing a sequence of evolutionary prototypes which are reviewed with 
the customer, both to ensure that the system is developing toward the user's requirements and 
to discover further requirements. The process is controlled by restricting the development of 
each integration to a well-defined period of time, called a time box. Each time box includes 
analysis, design, and implementation of a prototype. 


4: The exploratory model 


In some situations 1t is very difficult, 1f not impossible, to identify any of the requirements for 
a system at the beginning of the project. Theoretical areas such as Artificial Intelligence are 
candidates for using the exploratory model, because much of the research in these areas is 
based on guess-work, estimation, and hypothesis. In these cases, an assumption is made as to 
how the system might work and then rapid iterations are used to quickly incorporate 
suggested changes and build a usable system. A distinguishing characteristic of the 
exploratory model is the absence of precise specifications. Validation is based on the 
consistency of the end results and not in compliance with existing requirements. 


8. The spiral model 


The spiral model is similar to the incremental model, with more emphases placed on risk 
analysis. The spiral model has some resemblance to Deming’s “Plan, Do, Check, Act” cycle 
and has four activities: Planning, Risk Analysis, Engineering and Evaluation (Cf. Figure 1.8). 
A software project repeatedly passes through these activities in iterations (called Spirals in 
this model). In the planning activity, requirements are gathered and risk is assessed. In the risk 
analysis activity, a process 1s undertaken to identify risk and alternate solutions. A prototype 
is produced at the end of the risk analysis activity. Software is produced 1in the engineering 
activity, along with festing at the end of the activity. The evaluation activity allows the 
customer to evaluate the output of the project to date before the project continues to the next 
spiral. 


In the spiral model, the angular component represents progress, and the radius of the spiral 
represents cost. 


Risk analysis 


Engineering 
Figure 1.8 — Spiral development model 
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9. Common framework between software development life cycles 


In conclusion, there are a lot of models and many companies adopt their own, but all have 
very similar patterns. The general, basic model is shown in Figure 1.9: 


Figure 1.9 — General software development life cycle model 


Each activity produces deliverables required by the next activity in the life cycle. 
Requirements are translated into design. Code is produced during implementation that 1s 
driven by the design. Testing verifies the deliverable of the implementation activity against 
requirements. 


a. Requirements analysis 


Customer requirements are gathered in this activity. Meetings are held in order to determine 
the requirements. Who is going to use the system? How will they use the system? What data 
should be input into the system? What data should be output by the system? These are general 
questions that get answered during a requirements gathering activity. This produces a list of 
functionality that the software product must provide. 


b. Design 


The software system design is produced from the results of the requirements analysis activity. 
In this activity, the details on how the system has to work are produced. Architecture 
(including hardware and software), communication and software design are all part of the 
deliverables of a design activity. 


c. Implementation 


Code is produced from the deliverables of the design activity during implementation. 
Implementation may overlap with both the design and testing activities. Many tools exist to 
actually automate the production of code using information gathered and produced during the 
design activity. 


d. Testing 


During testing, the implementation is tested against the requirements to make sure that the 
product is actually solving the needs addressed and gathered during the requirements analysis 
activity. Unit, integration and validation tests are done during this activity. Unit tests act on a 
specific module of the system, while integration and validation tests act on the system as a 
whole. 


GC Two complementary approaches to design “bug-free” software 


Let us start this section by giving a definition for the term bug. In this dissertation, we adopt 
the definition proposed by /EEE. 
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Definition 1.4: Mistake, error, fault, failure/bug (EEE Std. 610-1990) 

“Mistake” is a human action that produces an incorrect result. 

“Error” is a difference between a computed result and the specified or theoretical one. 
“Fault” is a defect in a module which is the manifestation of an error. 

“Failure” is the inability of a system to perform a required function within specified limits. 


The relation between a mistake, an error, a fault and a failure is illustrated in Figure 1.10. 


Mistake — Error —> Faut —— Failure/bug 


Figure 1.10 — Relation between a mistake, an error, a fault and a failure 


The results of a software testing activity is a failure, therefore an analysis activity is 
necessary to identify the fault. 


In our research, we used the term “bug” rather than “failure” 


Software’s complexity and accelerated development schedules make designing “bug-free” 
software difficult. In this dissertation, we also adopt the definition of software quality 
proposed by EEE. 


Definition 1.5: Software quality (IEEE Std. 610-1990) 
(1) The degree to which a system, module, or process meets specified requirements. 


(2) The degree to which a system, module, or process meets customer or user needs or 
expectations. 


A widely accepted premise on software quality is that software is so complex (in 
combinatorial terms) that it is impossible to have “bhug-free” software. One technique 
commonly used in industry to verify and validate a software product is the software testing. In 
this dissertation, we adopt the definition of software testing proposed by the National Institute 
of Standards and Technology"' (NIST). 


Definition 1.6: Software testing and execution (NIST 2002) 


Software testing is the process of applying metrics to determine product quality. Software 
testing is the dynamic execution of software and the comparison of the results of that 
execution against a set of pre-determined criteria. “Execution” is the process of running the 
software on a computer with or without any form of instrumentation or test control software 
being present. “Predetermined criteria” means that the software’s capabilities are known 
prior to its execution. What the software actually does can then be compared against the 
anticipated results to judge whether the software behaved correctly. Software testing is a 
widespread V&V technique in automotive industry. 


In Chapter 8 — Section 2, We demonstrate that the software testing problem is a NP- 
Complete” problem. We often hear maxims like "there's always one more bug", and 
"software V&V techniques can reveal the existence of bugs, but never prove their absence”. 


!! http://www. nist.gov/ (Consulted on November 2008) 
NP: Non-deterministic Polynomial time 
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The Figure 1.11 released by Liggesmeyer (Liggesmeyer 1998) shows that the main part of 
bugs are introduced during the first life cycle of the software development (around 90% in 
requirements analysis, design and implementation activities) and detected in the last activities 
(around 80% during unit test, validation test and serial production). À study done in 2006 by 
a Johnson Controls software expert (Le Corre 2006) on about 15 projects from different types 
of products has confirmed the findings of Liggesmeyer. 


Detected ; 
bugs (in %) ." 


Introduced 
bugs (in %) 


Time (non-linear) 


Figure 1.11 — Rate and cost of bugs introduced and detected across the software 
development life cycle (Liggesmeyer 1998) 


Therefore, appropriate techniques, methods and procedures must be adopted in order to help 
engineers to: 


e lower the number of bugs introduced in the software system (prevention approach), 
e and detect all the bugs that have been introduced in the software system as soon as 
possible (detection approach). 


1. Prevention approach (Mays 1990, Gale 1990, McDonald 2007) 


Bugs are a consequence of the nature of human factors in the designing task. They arise from 
oversights made by engineers during requirements analysis, design, implementation and even 
testing. Complex bugs can arise from unintended interactions between different parts of the 
software system. This frequently occurs because software systems can be complex - millions 
of lines long in some cases - often having been programmed by many people over a great 
length of time, so that engineers are unable to mentally track every possible way in which 
parts can interact. The software industry has put much effort into finding methods for 
preventing engineers from inadvertently introducing bugs while designing a software system. 
These methods include: 


Engineers practices 
Standards 

Formal languages 
Prototyping 

Modeling and simulation 
Reuse 
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e Root cause analysis 


Even with efficient bugs’ prevention techniques and taking the assumption that human is not 
perfect, we conclude that each software system includes bugs. That is why, verifying and 
validating the software system before any customer delivery is a necessary activity. 


2: Detection approach: V&V techniques (Beizer 1995, Myers 1978, So 2002) 


Finding and fixing bugs has always been a major part of designing software systems. Maurice 
Wilkes”* , an early computing pioneer, said in the late 1940s (Wilkes 1949) that much of the 
rest of his life would be spent finding errors in his own programs. As computer programs 
become more complex, bugs become more common and hard to fix. Often programmers 
spend more time and effort finding and fixing bugs than writing new code. Theoretical data 
proposed by Brooks * (Brooks 2007) but also the results of a study done in 2006 by a Johnson 
Controls software expert (Le Corre 2006) on about 5 projects from different types of products 
are presented in Table 1.1. The study points out that the validation test activity takes up to 
50% of the total development duration of a project. Moreover, a recent study within Johnson 
Controls (2008) has shown that this ratio number has been exceeded. 


Software life Johnson F. Brooks Hewlett- 
cycle Controls (%) (%) Packard (%) 
40 33 37 


Requirements 


analysis 

Design and 

17 

implementation 20 34 
Unit test 25 

Integration and 40 05 29 


validation test 


Table 1.1 - Time and effort spent during the software development life cycle (Brooks 
2007, Le Corre 2006) 


Usually, the most difficult part of finding a bug is locating the erroneous part of the source 
code. Once the error is found, correcting it is usually easy. Programs known as debuggers 
exist to help programmers locate bugs. However, even with the aid of a debugger, locating 
bugs is something of an art. It is not uncommon for an error in one section of a program to 
cause bugs in a completely different section, thus making it especially difficult to track. 
Typically, the first step in locating a bug is finding a way to reproduce it easily. Once the bug 
is reproduced, the programmer can use a debugger or some other tool to monitor the 
execution of the program in the faulty region, and find the point at which the program went 
astray. It is not always easy to reproduce bugs. Some bugs are triggered by inputs to the 
program which may be difficult for the programmer to re-create. 


Therefore, bugs’ detection is still a tedious task requiring considerable manpower. Since the 
1990, particularly following the Ariane 5 Flight 501 disaster, there has been a renewed 
interest in the development of effective automated aids to remove bugs but it’s still remaining 
much of a work in progress. Presently, bugs” detection techniques (also called software V&V 
techniques) can be classified into two classes: 


5 http:/www.cl.cam.ac.uk/-mvwl1/short-biography.html (consulted on November 2008) 
* Frederick P. Brooks is a pioneer of software engineering, http:/www.cs.unc.edu/-brooks/ (consulted on 
November 2008) 
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e Static techniques (Review and Proof) which do not require the execution of the 
software under test (Ayewah 2008) 

e Dynamic techniques (Testing) which require the execution of the software under test 
(Beïzer 1990, Barezi 2006). 


Each of these techniques catches different classes of bugs at different points in the 
development cycle. 


D. Impacts of detecting bugs later in the software development life cycle 


According to a newly released study commissioned by the Department of Commerce's 
National Institute of Standards and Technology (NIST 2002), software bugs cost the U.S. 
economy an estimated $59.5 billion annually, or about 0.6 percent of the gross domestic 
product. The study also found that, although all errors cannot be removed, more than a third 
of these costs, or an estimated $22.2 billion, could be eliminated by an improved V&V 
infrastructure that enables earlier and more effective identification and removal of software 
bugs. These are the savings associated with finding an increased percentage of bugs closer to 
the development activities in which they are introduced. Currently (Cf. Table 1.1), over half 
of all errors are not found until the last testing activity in the development process (validation 
test) or during post-sale software use (operational life). 


The impact on the software industry due to lack of robust, standardized V&V technology able 
to detect bugs closer to where they are introduced can be grouped into three general 
categories: 


1; Poor quality perceived by the customer 


The most troublesome effect of a lack of efficient V&V technology is the increased incidence 
of avoidable bugs that emerge after the product has been delivered to the customer. Poor 
quality often results in loss of reputation and loss of future business for the company. In 
addition, legal actions are undertaken against the supplier when bugs are attributable to 
insufficient V&V. 


2, Increase of the software development cost 


Historically, the process of identifying and correcting bugs during the software development 
process represents over half of development costs. Depending on the accounting methods 
used, V&V activities account for 30 to 90 percent of labor expended to produce a working 
program (Beizer 1990). Software engineers already spend approximately 50 percent of 
development costs on 1dentifying and correcting bugs (Cf. Table 1.1). Early detection of bugs 
can greatly reduce costs. Bugs can be classified by where they were found or introduced along 
the activities of the software development life cycle, namely, requirements analysis, design, 
implementation, testing, and operational life activities. Figure 1.11 illustrates that the longer a 
bug stays in the program, the more costly it becomes to fix 1t. 


3. Increase of the time to market 


The lack of efficient V&V technology also increases the time that it takes to bring a product to 
market. Increased time often results in lost opportunities. For instance, a late product could 
potentially represent a total loss of any chance to gain any revenue from that product. Lost 
opportunities can be just as damaging as post-release product bugs. However, they are 
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notoriously hard to measure. If efficient V&V techniques were readily available, engineers 
would expend less time developing custom V&V technology. 


VI. Industrial needs and expectations 


Nowadays, electronics represents more than 30% of the global cost of a car (Sangiovanni- 
Vincentelli 2003). Car electronic architecture becomes more and more complex and 
carmakers outsource the design of electronic modules to automotive electronic suppliers. The 
software part is the added value of these modules and they account for more than 80% of the 
total number of problems detected on these modules (Johnson Controls source). As 
automotive electronic products become more and more complex, the size of software 
embedded in these products increases drastically. In fact, a body controller module managing 
the interior function of a car body account for more than 200 KLOC” (Johnson Controls 
source). As a consequence, the time spent in verifying and validating these software has 
increased exponentially the last 10 years. V&V activities account now for more than 50% of 
an automotive electronic project time and effort (Cf. Table 1.1). Despite the huge resources 
spent in verifying and validating a software product and after each delivery to the carmaker, 
some bugs are detected by the carmaker and forwarded to the supplier who must react quickly 
and efficiently. Once an electronic module is launched on the market (e.g. integrated into a 
vehicle), an average of one software bug per year 1s detected by the end-users, which may 
becomes dramatic for the electronic supplier in financial terms if the product has to be 
systematically changed. In fact, in term of bug’s occurrence, two types of contract engage 
electronics suppliers with car manufacturers: 


e _Implicit contract: during software development process, each carmaker delivery must 
be free of bugs. 

e Explicit contract: on launched electronic module, carmaker tolerates a certain number 
of defective products expressed in terms of PPM (Pieces Per Million). PPM includes 
all software bugs but also electronic, mechanical and production problems. 


As the automotive market becomes more and more competing, decreasing the development 
me of outsourced parts and decreasing the number of problems detected later in the process 
becomes of major importance for carmakers and consequently a major quality indicator for 
automotive suppliers. Indeed, the carmakers” process for assigning new projects to suppliers 
is mainly based on feedbacks from previous projects. Consequently, suppliers work on 
reducing the development time of their products, delivering on time the products to carmakers 
and detecting the maximum number of bugs as earlier as possible in the development process. 


Through our research project, we were asked by an automotive electronic supplier namely 
Johnson Controls to improve the performance of its software V&V activities. Their main 
purpose is to improve the quality of their products and therefore better satisfy the 
requirements and expectations of their clients. In Johnson Controls, the software development 
life cycle follows a V-model (Cf. Chapter 2 — Section 3.B). Moreover, the validation test 
which is the last V&V activity before a carmaker delivery 1s considered as the ultimate activity 
to detect all the bugs and therefore deliver carmakers “bug-free” software. It represents up to 
90% of the time spent in the V&V of a software product (Johnson Controls source). Testing a 
software product requires two main activities. À detailed specification of these activities 1s 
done in Chapter 2 — Section 5. The first one consists of designing test cases and the second 


15 KLOC : Kilo Lines Of Code. 
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one of executing these fest cases on the software product under test. We adopt the definition 
of test case proposed by /ÉEE. 


Definition 1.7: Test Case (IEEE Std. 610-1990) — Abbreviation: TC 
IEEE defines test case as follows: 


(1) À set of test inputs, execution conditions, and expected results developed for a particular 
objective, such as to exercise a particular program path or to verify compliance with a 
specific requirement. 


(2) Documentation specifying inputs, predicted results, and a set of execution conditions for a 
test item. 


While the execution activity 1s often automated via a test execution platform, the test case 
design activity remains a manual task that fills in time many engineers. We propose below our 
definition of a test execution platform. 


Definition 1.8: Test execution platform 


À test execution platform is a platform that aims to simulate the environment of and perform 
test inputs on the software module or product under test. Therefore, one can verify and 
validate the behavior of the software under test in its real environment. For more information 
regarding the Johnson Controls test execution platforms, please refer to Appendix C. 


Up to 50% of a software project time 1s dedicated to design rest cases (Cf. Table 1.1). 
Therefore, for many of software managers and experts in Johnson Controls, automating the 
design of test cases seems to be the most adapted solution to reduce the testing time, cost and 
resources while improving the code and requirement coverage. We adopt the definition of 
coverage proposed by EEE. 


Definition 1.9. Coverage (1EEE Std. 610-1990) 


In software engineering, the term “coverage” means the degree, expressed as a percentage, 
to which a specified coverage item (code or requirement) has been exercised by a test case. 
For the definition of code (structural) and requirement (functional) coverage, please refer to 
Chapter 2 — Section 5.F. 


In our research, we go through this problem with a systemic approach in order to identify 
levers in any domains from which we might be able to improve the global performance of the 
software V&V activities. The added value of such an approach is the resolution of the problem 
with a global quality viewpoint. Consequently, in Chapter 2, we characterize the software 
design environment in automotive industry and point out issues and anomalies (diagnoses). In 
Chapter 3, based on our industrial audit, We clearly define the scope of our research and we 
formulate our research topic in accordance with the research issues in software testing. In 
Chapter 4, We perform a literature review on the existing approaches, techniques and tools in 
the field of the V&V of software products. More especially, we focus our research on finding 
or adapting “solutions” for the anomalies and lacks (diagnoses) that we identify via our 
industrial audit. We identify relevant actions for improving the global performance of the 
Johnson Controls V&V activities. In Chapter 5, 6, 7 and 8, we specify our proposed models. 
A prototype implementing our models has been developed in Chapter 9. Finally, in Chapter 
10, we validate our models through two industrial case studies on historical data. 
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VIL. Conclusion 


Presently, automotive industry is facing significant difficulties in terms of selling new cars. 
Therefore, carmakers ask their suppliers to innovate, increase quality, reduce time to market 
and decrease the development cost. As electronics represents more than 30% of the global 
cost of a car and stands for a big amount of the problems detected on a car, electronic 
suppliers are the most concerned. Software technology is at the core of each electronic 
product; therefore, electronic suppliers focus their efforts on the improvement of their 
software development and V&V practices. Through our research project, we were asked by an 
automotive electronic supplier namely Johnson Controls to improve the performance of its 
software V&V activities. Their main purpose is to better satisfy the requirements and 
expectations of their clients in terms of quality, cost and delay. In our research, we go through 
this problem with a systemic approach in order to identify domains from which we might be 
able to improve the global performance of the Johnson Controls software V&V activities. 


In the following chapter, we perform an industrial audit on the software practices and more 
especially on the V&V techniques currently used in automotive industry. We aim to identify 
the issues and lacks of the current practices in order to propose relevant improvement actions 
well adapted to the industrial context. 
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CHAPTER 2. INDUSTRIAL AUDIT 
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I. Introduction 


The audit of the industrial context permits to identify and determine the overall environment 
in which our research project has to be performed. This must results in a better understanding 
of what verifying and validating a software product means and what are the necessary changes 
to perform. 


In this chapter, we perform an industrial audit on the software practices currently used in 
automotive industry and more especially in Johnson Controls. The audit is divided into four 
parts: 


e The process of managing the carmakers’ requirements related to the software domain. 
In fact, delivered software products must be compliant with the carmaker’s 
requirements. 

e The processes of verifying and validating software products. Many Verification and 
Validation (V&V) activities are performed on a software product before the delivery to 
the carmaker. 

e The process of managing and reusing capitalized bugs. Indeed, bugs detected on 
previous projects and stored in the problems’ database must be regularly reviewed in 
order to avoid similar bugs on new developments. 

e The process of managing and reusing capitalized fest cases. In automotive industry, 
the projects related to the same type of product and car platform of one carmaker have 
up to 70% of common functionalities (Johnson Controls source). Therefore, reusing 
test cases from one project to another must be done frequently. 


For each of these parts, we make our analysis on two stages: 


1. À snapshot of the current software practices in Johnson Controls (process, tool, 
people) 
2. Analysis and diagnoses of these practices. 


In the conclusion of this chapter, we summarize the performed diagnoses and we locate them 
within the Johnson Controls software organization. 


IL. Frame of the audit 


Our approach to audit the practices currently used in Johnson Controls when verifying and 
validating software products can be divided into 7 activities: 


e  Analyze the documents delivered by the carmakers to their electronic suppliers. Their 
formats and their evolutions during the software development life cycle 

e  Analyze the main activities of an engineer when designing test cases for a software 
product. This analysis is performed with a multi point of view: process, tool, and 
people 

e Audit engineers when designing rest cases 

° Intervention on the design of fest cases for four software projects 

e Interview managers on the expectations of the carmakers at each stage of the software 
development life cycle 

° Interview all types of engineers that can be involved in a software project 

e Analyze data on the software testing practices of carmakers 


In the following, the results of the audit are presented. 
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III. The software projects in automotive industry 


A. An incremental software development process 


Definition 2.1: Project (Wikipedia —- November 2008) 


A project is a temporary endeavour undertaken to create a unique product or service. If can 
also comprise an ambitious plan to define and constrain a future by limiting it to set goals 
and parameters. The planning, execution and monitoring of major projects sometimes 
involves setting up a special temporary organization, consisting of a project team and one or 
more work teams. 


A project consists of a set of coordinated and controlled activities organized to achieve an 
objective conforming to specific requirements. Presently, at Johnson Controls, a product 
project (hardware, software and mechanical skills) typically represents 24 months of 
development and involves around 25 engineers (Johnson Controls source). The five main 
stages of a project are 1llustrated in Figure 2.1 and described in Table 2.1. Each stage is 
defined by a procedure that identifies the responsibility, deliverables (inputs, processes and 
outputs) and applicable references. The main focus of these stages 1s the design and testing of 
components and assemblies through product launch. It is a customizable process that is to be 
applied for all automotive products within Johnson Controls company. 
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Figure 2.1 - Product development system at Johnson Controls - Some parts of this figure 
are voluntarily fuzzyfied for confidentiality reasons (Johnson Controls source) 


Phase 1:Proposal Responding to customer inquiries regarding new products. 
The requirements of the customer are identified and 
proposals are submitted for customer approval 


Phase 2:Design & Expanding upon the proposal through the establishment of 
Development the product definition to ensure productfeasibility 


Phase 3: Design Verification  Completing product design resulting in a detailed definition 
of both the product and process 


Phase 4: Production Encompassing the activities required to ensure that the 
Validation product meets all customer requirements when produced 
Phase 5:Launch Ensuring production preparation to ensure a smooth 


transition from production initiation to volume production 


Table 2.1 — Description of the stages of a product project (Johnson Controls source) 
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The objective of the milestones of the product development system is to check the status and 
the progress of the program. Software skill has to take into account the hardware and 
mechanical releases availability and carmaker deliveries requirements of the product project 
to define its life cycle and its planning. In Figure 2.1, the steps of the high level software life 
cycle are mapped with the product development stages. A detailed description of the software 
steps is given in Table 2.2. 


SOFTWARE Description 
STEPS Ê 


Proposal Analyze the customer need in order to estimate workload, schedule and resources 
needed to perform the software project 


Preparation Analyze in detail main customer requirements and deliver a 1st prototype (functionality 
level) 


Analysis & Design Analyze all customer requirements, define details of software architecture, deliver a 
functional release that contains main customer requirements and define the validation 
strategy 


ED* Development Complete the analysis of remaining requirements and deliver a partially functional 
release, with full functional but without diagnostic 


DV* Development Complete the global design, deliver a partially functional release without validation and 
manufacturing functionalities but with full functional and diagnostic and complete the test 
report for DV release 


PV*Development Complete the analysis of remaining requirements including validation and manufacturing 
requirements, specify and accept testing tools for production line 


Testing Perform a full testing of the software, improve the software reliability from customer tests 
feedback and complete the test report for PV Release 


Adjustment Take into account last minute customer changes and possible software problems and 
deliver an industrial release 


Maintenance Take into accounteventual problems on serial products and customerchanges 
* ED: Engineering Development 


DV: Design Verification 
PV: Product Validation 


Table 2.2 — Description of the steps of the high level software life cycle (Mignen 2006a) 


The software life cycle is initialized during the Proposal step and adjusted during the 
Preparation step according to carmaker deliveries planning and requirements prioritization 
(Cf. Figure 2.1). 100% functional software means that all functionalities are implemented 
(carmaker and manufacturing requirements are implemented). If significant changes have to 
be implemented for a new release during the Maintenance step, a new software project 1s 
launched and the program remains in stage #5. 


B. The elementary V-model of the software development process 


Within each step of the software standard life cycle, engineering activities are performed 
according to the standard V-model of the software industry (Cf. Figure 1.6) and in an 
incremental Way in order to take the carmaker constraints and requirements priorities into 
account. The number of incrementations per step is defined by the project and adjusted based 
on carmaker inputs and project constraints. Based on the SPICE'® model, Johnson Controls 
has developed a process map implementing the conventional V-model (Cf. Figure 2.2). 


1 SPICE: Software Process Improvement and Capability dEtermination 
(http:/www.sqi.gu.edu.au/spice/contents.html, Consulted on November 2008). 
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[Validation | 
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| _ GlobalDesign | | Integration | 


| Component Development | 


Support activities 


Figure 2.2 — Process map implementing the software V-model at Johnson Controls 
(Mignen 2006a) 


Each process or group of processes (Support processes) of the process map is synthetically 
described in Table 2.3. 


SOFTWARE PROCESS 


Project Management (PM) Plan and monitorthe software project 
Manage risks and documentation 


Requirements Specification (RS) Define, classify and prioritize requirements 
Establish traceability after validation 


Global Design (GD) Define general software architecture, components andinterfaces 
Provide traceability and verify global design 


Component Development (CD) Develop detailed design, produce and verify components 
Develop, review andexecute unittest procedures 


Integration (INT) Define software integration strategy 
Perform incremental integration and execute integration tests 


Validation (VAL) Develop software validation strategy 
Design, implement and perform validation procedure 


Supportprocesses Control changes to configuration items 

(Not detailed for confidentiality reasons) Plan, track, verify and validate changes / defects 
Perform document and projectreviews 
Perform Software quality and process audits 


Table 2.3 - Description of the software processes within Johnson Controls (Mignen 
2006a) 


As illustrated in Figure 2.1, these software processes are carried out before each carmaker 
delivery of the software product. Despite the V&V of the software product and after each 
delivery to the carmaker, some bugs are detected by the carmaker. This could lead to the 
conclusion that carmakers have more efficient testing approaches than their suppliers. But, 
carmakers do not communicate on their practices and furthermore, they do not often transmit 
test cases to their suppliers. We analyze data on the software testing practices of carmakers 
and interview inner experts in touch with the carmakers. As a conclusion, this efficiency in 
testing Software products can be related to many factors such as: 
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e The carmakers benefit from real electronic platforms where the supplier modules are 
installed and tested. In fact, the modules are tested in a simulated real environment 
with surrounding modules in a global system approach. 

e The carmakers use their experience feedback of recurrent bugs to test a given module 
with the knowledge of bug probabilities and even end-user behavior’s profiles to 
design the most relevant fest cases. 


Diagnosis 1 


Verification and Validation practices and test cases are rarely shared between the 
carmakers and their electronic suppliers. 


GC: Functional organization of a software project 


Besides the project leader, the coordination team and the quality team, one can identify two 
technical teams in a software project at Johnson Controls (Cf. Figure 2.3): 


e One in charge of the development of the software product 
° And the other in charge of its validation. 


Project leader 


Coordination team Quality team 


Validation 
Development team 


team 


Figure 2.3 - Typical functional organization chart of one software project at Johnson 
Controls - Not detailed for confidentiality reasons (Mignen 2006b) 


The coordination team is located in so called “front office” sites, close to the carmakers. 
Development and validation teams are in general located in Low Cost Countries (LCC). 
However, in some cases, they can be spread across several locations. Globally, we have 
Software Developer (SD or developers) who develop the software product, Software 
Validation Engineer (SVE or validator) who validate the software product before a carmaker 
delivery, Software Coordinators (SC) who are responsible for the assignment of the quality, 
schedule, cost goals and quality engineers Who ensures that project quality commitments are 
respected. For confidentiality reasons, we are not allowed to give more details on the roles 
within a software project. 


Ten years earlier, software V&V was covered only in software engineering courses. 
Nowadays, American but also European universities have responded to the importance of the 
V&V practices in industry with new independent courses and specialties in verifying and 
validating software products (Duernberger 1996). The goal of these courses is to prepare 
students for software testing management, testing considerations, designing test cases, and 
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applying various testing tools and methodologies. The fact remains that industrials give value 
to software verification and validation roles. In fact, software engineers often prefer 
development activities compared with verification and validation activities. This observation 
has been confirmed after interviewing software managers within Johnson Controls. 


Diagnosis 2 


Now, one cannot get a degree in software V& V. Software V&V is incorporated into the 
software engineering degree. Moreover, software engineers often prefer development 
activities compared with verification and validation activities. 


IV. Management of the carmaker requirements related to software 


Let us start by defining a common vocabulary on carmakers’ requirements related to the 
software domain. In this dissertation, we consider the definition of specification and 
requirement proposed by ZEEE. 


Definition 2.2: Specification (IEEE Std. 610-1990) 


À specification is a document that specifies, ideally in a complete, precise and verifiable 
manner, the requirements, design, behavior, or other characteristics of a component or 
system, and, often, the procedures for determining whether these provisions have been 
satisfied. 


Definition 2.3: Requirement (IEEE Std. 610-1990) 


À requirement is a condition or capability needed by a user to solve a problem or achieve an 
objective that must be met or possessed by a system or system component to satisfy a contract, 
standard, Sspecification, or other formally imposed document. There are two groups of 
reguirement: 


- Functional requirement: À requirement that specifies a function that a component or system 
must perform. 


- Non functional requirement: À requirement that does not relate to functionality, but to 
attributes such as reliability, efficiency, usability, maintainability and portability. 


We also adopt some definition proposed by Johnson Controls software experts. 


Definition 2.4: (Software) Functionality (Johnson Controls) 


À functionality (called also client or software functionality) is described by some features that 
are described by some requirements. For instance, a speedometer is a functionality of a 
cluster 


The term “Function” is not used in reguirement management to avoid misunderstandings 
with the coding language. 


Definition 2.5: Feature (Johnson Controls) 


A feature is a “property” or “behavior” of a software. It describes the particularity of a 
device. Each feature is composed from one or more requirements. For instance, a 
“Speedometer” is a feature of a cluster. It can be broken down into 3 features: 


- Speed display 


- Speed computation 
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- Conversion from Km to miles 


The breakdown granularity has to be adjusted according to the project needs. 


Definition 2.6: Requirement (Johnson Controls) 


A reguirement is something to be done to design the device (it is required). For instance, the 
value of the speed shall be calculated using the X data. Each requirement must contain a 
subject / object and a predicate: 


- Subject = system, user 
- Verb / Predicate = action 


The whole Requirement needs to define a result. À performance or measurable indication 
needs to be included 


Example: 
The display backlight has to be switched on in less than 1 second after ignition on, 
A ï a  — . 


ns 
object action result performance/measurable 


There are 5 types of requirements: 

- FCT: functional requirement (behavior of the software) 

- CON: constraint requirement (reliability, safety, quality, process, rules, guidelines ….) 

- INT: interface requirement (specification of internal and external interfaces) 

- DEV: development requirement for internal development support (interface, parameter ….) 


- MMI: man-machine interface requirement (Menus, buttons ….) 


Within the customer requirements, the software functional requirements account for 
more than 90% (Johnson Controls source). We also validate this proposal by analyzing 
the carmaker requirements for 5 different software projects (different products) in 
Johnson Controls. Therefore, through our research project, we focus on the software 
functional requirements and how one could verify the compliance of a software product 
with its functional requirements. 


A. The carmakers specification of software functional requirements: 
diversity, typology, evolution 


At the beginning of a project, automotive suppliers officially receive the electronic product 
requirements from the carmakers. In fact, there is no standardization between carmakers and 
electronics suppliers on the way requirements must be expressed. Based on this deliverable, 
suppliers analyze and identify skill requirements (soffware, hardware and mechanical). 
Afterwards, software requirements are sorted by the software department according to the 
typology proposed in Definition 2.5. In our research, we focus on the software functional 
requirements on which we identify two main characteristics: 


1. Carmakers consider different standards to express the software functional 
requirements of a given electronic module. Some carmakers use semi-formal methods, 
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such as Sratechart or UML"? illustrated respectively in (Harel 1987) and (OMG 2005), 
others use natural language. Even, one carmaker could use two or more different 
standards according to each department policy. In 2007, we carried out a study on the 
evolution of the formalisms used by carmakers to specify software functional 
requirements. In Chapter 4 — Section 4.D.1, we do a survey on the formalisms used to 
specify the functional requirements of a software product (Dart 1987, Brinkkemper 
1990). Three levels of formalism have been identified: informal, semi-formal and 
Jormal. The study was done on eight editions of carmaker requirements documents 
spanning from 1997 till 2006. We also considered three different carmakers: two 
Europeans and one Japanese but only one type of electronic product. The results of the 
study are illustrated in Figure 2.4. We underline the increase of formal methods based 
on the requirements simulation and the decrease of informal and semi-formal methods. 
Since this conclusion is fully true for the considered type of product, it is partially true 
for other types of product where natural language and semi-formal methods are still 
widely used. However and according to automotive experts, the trend is toward formal 
methods. Through this study, we also noted that software functional requirements are 
often expressed in many documents, emails, and even some phone calls. 


% of the formalism 


3 Informal 


“ Formal 


$ Semi-formal 


LU 


RQ 


DO 


= LLODRRQMWL 


1997 1999 2001 2003 2005 2006 2006 2006 
Edition date of the software functionalrequirements 


Figure 2.4 — Evolution of the formalisms used by carmakers to specify the functional 


requirements related software 


UML: Unified Modeling Language (http://www.uml.org/, consulted on November 2008). 
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Diagnosis 3 


In automotive industry, semi-formal and formal methods are more and more used to specify 
software functional requirements. However, there is a lack of a standard formalism shared 
between carmakers and suppliers. In fact, for each project, the supplier has to adapt its 


processes to the formalism used by the carmaker. 


2 


Software functional requirements continuously evolve during development phases and 
also during operational life of the product. In 2007, we analyze the evolution on one 
project of the carmaker requirements related to software. The studied project started in 
2005 and it is considered (by experts) as a typical project in automotive electronics 
industry. The Figure 2.5 illustrates the results of the study. We note the growth 
number of changes asked by the carmaker after the date of software requirements 
freeze. We interview inner software experts and managers, often in contact with the 
carmakers, in order to understand this phenomenon. In fact, at the beginning of a 
project, the carmaker does not have a 700%-clear view of what each functionality 
should perform. It is through the project and after each delivery that the expected 
behaviors become clearer. Moreover, suppliers are often more experimented in the 
development of automotive electronic products than some carmakers (system 
integrator). This leads to the fact that some carmakers lean on the suppliers by letting 
them identify inconsistencies and ambiguities in the product specifications. 


70 - 66 
—— Cumulated number of the 
60 « software functionalities 
53 653 53 53 53 implemented in the delivery 
49 8 —#} 
50 - 
41 
40 - Number of the requirements 
change requests done by 
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Ê) 
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Date of deliveries jan-05 mar-05 juin-05 sept-05  jan-06 juin-06 sept-06 jan-07 ma-07 
Carmaker software requirements Fos Roc co4 ane unes des 
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Cumulatednumber of the software 

functionalities implemented in the 9 24 a1 49 53 53 53 53 53 
delivery 

Numberofthe requirements change 

requests done by the carmakeronce 0 1 1 20 13 66 14 0 0 


the date of freeze is passed 


Figure 2.5 — Growth of the number of changes asked by the carmaker all along a project 


Diagnosis 4 


Deadlines for carmaker requirements freeze are specified in the carmaker-supplier 
contract. Nevertheless, the carmaker’s requirements evolve continuously along the software 
development life cycle without complying with these deadlines. Moreover, suppliers must 


react quickly by integrating (without regression) the changes in the product. 
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B. Commitment contract between carmakers and electronic suppliers 


As noticed before, a loop-type design process is initiated between the carmaker and the 
supplier. About ten intermediary client deliveries are carried out. After each delivery, some 
“bugs” are detected by the carmaker and forwarded to the supplier who must react quickly 
and efficiently. Once an electronic module is launched on the market (e.£. integrated into a 
vehicle), an average of one “bug” per year is detected by the end-users, which may becomes 
dramatic for the electronic supplier in financial terms if the product has to be systematically 
changed. 


We analyze typical contractual documents between carmakers and electronic suppliers. 
Moreover, we interview Johnson Controls managers in charge of establishing these contracts. 
In fact, in term of bugs’ occurrence, two types of contract engage electronics suppliers with 
car manufacturers: 


e Explicit contract: on launched electronic module, carmakers tolerate a certain number 
of defective products expressed in terms of PPM (Pieces Per Million). PPM includes 
all software bugs but also electronic, mechanical and production defects. For instance, 
in Table 2.4, when starting production (SOP) the carmaker tolerates X PPM on 0 km 
cars. It 1s logical that the required number of PPM on 0 km cars decreases (Y<X) 4 
months after the production has started. The number of PPM is negotiated at the 
beginning of a project. The electronic supplier estimation of their capability in term of 
PPM number is mainly based on the experience feedback but also on the product 
complexity and novelty. 


{so | SOP +4 months | SOP + 1 year 


0 km X ppm** Y< x ppm Z<Y ppm 
3 months 

1 year 

3 years 


*SOP: Start Of Production 
*bpm:piece per million 


Table 2.4 - Explicit contract, in terms of bugs’ occurrence, between a carmaker and an 
electronic supplier 


e  Implicit contract: implicit aspects are usually disclosed later in the development life 
cycle and are generally based on semantic problems. For instance, during the software 
development process and even if it was not stated in the contract, the carmaker 
expected that each intermediate delivery must be free of bugs. Many other examples 
can be cited. In fact, the requirements specifications delivered by the carmakers are 
usually and purposely unclear and incomplete in order to be able to add, modify or 
remove one or more requirements. 


C. Sensitive criteria for carmakers 
Carmakers are sensitive to different criteria depending on whether the project is in its 


proposal, design and development or operational life phase. In order to identify these criteria 
for each phase, we interview 3 project leaders for 3 projects in each of these phases. 
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1. Proposal phase 


In proposal phase, carmakers choose their suppliers basically on economic criteria. An 
additional cost regarding other suppliers can exist but must be justified on quality and/or delay 
levels. Moreover, they strongly used their experience feedbacks on other or previous projects 
with each supplier. 


Presently, all the automotive electronic suppliers have almost the same knowledge and know 
how in the product design, development and maintenance. Therefore, competing with other 
suppliers on technical criteria remains very hard. 


Finally, the process improvement aspect becomes a major quality criterion for the carmakers. 
For instance, the carmakers require now that their suppliers have reached a specific maturity 
level within the SPICE or CMMI® models. These two models are process improvement 
approaches that provide organizations with the essential elements of effective processes. 


2. Product design and development phase 


During the product design and development phase, intermediate deliveries of the product 
(including all or part of the product functionalities) are planned. Carmakers are sensitive to: 


e Time to delivery: the supplier must respect the planning established at the beginning of 
the project. 

e _ Product quality: the supplier must test the product and validate its conformance with 
the carmaker requirements before the delivery. “Zero bug” is required by the 
carmaker. 

e Additional cost: sometimes, the supplier tries to invoice the modification or evolution 
requests asked by the carmaker. 


3. Operational life phase 


We identify three criteria to which carmakers are sensitive during the operational life of a 
product. We classify these criteria by priority order: 


e _ Regression risk While modifying or correcting the product: the financial impact on the 
supplier can be severe especially when the car product lines are stopped because of its 
product. In order to better 1llustrate this issue, let us consider the following example 
excerpted from a real situation. Once, a carmaker required a modification on a product 
in operational life phase. The modification as it was expressed by the carmaker was to 
“remove” a piece of software code from the product in order to avoid the hacking of 
the product and therefore the stealing of the car. The supplier has implemented this 
modification by erasing the piece of code, full validated and delivered the new product 
version. The carmaker has also made a full validation of the new version of the 
product. Unfortunately, when starting the serial car production and when integrating 
the product in the cars, a bug related to this modification has occurred and thus 
blocked all the car production lines. À deep analysis of the bug has revealed that the 
removed piece of code must not be removed from the product but hidden. One more 
example on the implicit requirements of the carmakers since the carmaker declared 
that when he asked for “removing the code”, he indirectly asked for “hide the code”. 


 CMMI: Capability Maturity Model® Integration (http://www.sei.cmu.edu/cmmi/, Consulted on November 
2008). 
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e Economic criteria regarding the modification and correction costs. 
e Time to deliver the new product version with the modifications and/or corrections 


requested. 
D. Snapshot of the Requirements Specification process at Johnson Controls 
1. Introduction 


The purpose of the current Requirements Specification process 1s to ensure that all software 
requirements reflect allocation of carmaker and/or system requirement to software are 
identified, documented, maintained, committed and validated to serve as a basis for software 
design, implementation and validation. As a result of the Requirement Specification process: 


e the software requirements to be allocated to the software components of the system 

and their interfaces are defined, 

software requirements are classified and analyzed for correctness and testability, 

the impact of software requirements on the operating environment is evaluated, 

prioritization for implementing the software requirements 1s defined, 

the software requirements are approved and updated as needed, 

consistency and bilateral traceability are established between system requirements and 

software requirements; and consistency and bilateral traceability are established 

between system architectural design and software requirements, 

° and the software requirements are baselined and communicated to all concerned 
people. 


2. Interfaces with other software processes 


The current Requirements Specification process is considered as the main important process 
within the software processes. In fact and as shown in Figure 2.6, this process strongly 
interacts with all other processes. Especially, it delivers the software requirements to allocate 
them to components (Global Design), to develop these components (Component 
Development) and to design associated test cases (Validation). 


Project Management / Documentation Management 


Requirement Specification 4 Validation J 
Fr 
ANA 


= 


Component 


Support processes 


(Not detailed for confidentiality reasons) 


207 <092Wm-3m%<WH-D230%EO 


Figure 2.6 — Interaction of the Requirements Specification process with the other 
software processes (Mignen 2008) 
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3. Process flow 


Within each step of the software standard life cycle (Cf. Figure 2.1), engineering activities are 
performed according to the process map illustrated in Figure 2.2 and in an incremental Way in 
order to take the carmaker constraints and requirements priorities into account. After 
analyzing internal documents related to the definition of the Requirements Specification 
process, we identify 6 main activities for the management of software requirements. In fact, 
for each design iteration, the Requirements Specification process flow follows the series of 
activities defined in Table 2.5. 
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Process Comments 


lteration start 


Carmaker documents of requirements are 


Elicit and maintain identified 
needs 


| The activity continues until all requirements in 
Define the scope of iteration are defined). A list is used 
requirements to track unclear needs until clarification with the 
carmaker 


All Software requirements are classified and 
SÉAUe rioritized 
prioritize req. P 


Define validation Special conditions for validating requirements are 
criteria defined 


Establish Traceability is established with carmaker and/or 
traceability system requirements 


AI the requirements are reviewed by internal and 


Obtain validation external concerned people. 
RCOMtIenr Commitment on software requirements is done 


with carmaker and system 


lteration stop 


Table 2.5 — Process flow of the Requirements Specification process 


a. Elicit and maintain needs 


This activity aims to establish and maïintain carmaker and system needs and expectation that 
will serve as a basis for specifying requirements allocated to software. The features required 
by the carmaker are identified and peer projects for these features are identified for use of 
lessons learned. 


b. Define requirements 


For each feature, software requirements are specified or updated using the Soffware 
Requirement Specification (SRS) model (see next section for the principles of this model). 
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Requirement Management tools such as Reqtify"” or Doors” can be used for managing and 
storing requirements. All items that need to be clarified are filled in a CLarification Request 
list (CLR) and discussed with concerned people. 


c. Classify and prioritize requirements 


For each software requirement identified in the SRS, one has to define: 


the type of the requirement (FCT, CON, INT, DEV, MMD), 

the status (new, accepted, confirmed, dropped), 

the priority (high, normal, low), 

the associated feature, 

and the required verification and validation technique (code analysis, code review, unit 
test, integration test, validation test). 


Other criteria can be added such as safety, revision and so on. A Requirement Management 
tool can be used for this classification. 


d. Define validation criteria 


In order to support the validation team and facilitate rest case definition, validation criteria 
need to be specified. These criteria consist of defining when the test can be considered as 
passed correctly (test case results acceptance). À validation criterion can be applicable on 
several requirements or group of requirements. Validation criteria can be based on a lesson 
learned. By default, standard validation criteria are the successfully passing of test cases (in 
this case, it 1s not necessary to define specific validation criteria). The validation criteria shall 
define special conditions for validating some requirements 1f their validation deviates from the 
standard use of tests cases. For instance: 


e Specific criteria to validate (respect of standard, performance criteria, different 
situations to validate ...). 

e Condition for validating the requirement (normal and specific conditions for the test, 
stress situations, tools, or negative tests). 

e Criteria defining when validation tests can be considered as passed correctly 
(including thresholds of performance deviation). 


e. Establish traceability 


Purpose of this activity is to establish and maintain upward bilateral traceability between user 
requirements (carmaker requirements, system requirements) allocated to software and 
software requirements in order to verify that all carmaker and system requirements that have 
been allocated to software are taken into account in the SRS. The result of this activity is a 
matrix called fraceability matrix. 


f. Obtain requirements validation and commitments 


Each time a step of the SRS elaboration is achieved in order to start a part of software 
development, the version of SRS is reviewed to make sure the understanding and commitment 


 http://www.geensys.com/?Outils/Reqtify (Consulted on November 2008). 
7 http:/www.telelogic.com/Products/doors/doors/index.cfm (Consulted on November 2008). 
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by the implementation and validation teams. Once the version of SRS is released, it is base 
lined and serves as a basis for implementation and validation activities. 


E. The Software Requirement Specification model currently used in 
Johnson Controls 


Automotive electronics suppliers like many software engineering organizations adopt the 
Software Requirement Specification (SRS) model to express the various expectations of a 
software product. À SRS model is a comprehensive description of the intended purpose and 
environment for software under development. The SRS fully describes what the software will 
do and how it will be expected to perform. À good SRS defines how an application will 
interact with system hardware, other programs and human users in a wide variety of real- 
world situations. Parameters such as operating speed, response time, availability, 
maintainability, security and speed of recovery from adverse events are evaluated. Methods of 
defining an SRS are described by ZEEE (IEEE Std. 830-1998). 


Johnson Controls has adapted the SRS model to its organization, needs and types of products. 
In Figure 2.7, the data model of the SRS currently used is described. The SRS document 
serves as a basis for software design and validation plan. 


The engineer responsible of managing the carmaker requirements shall update the SRS 
document according to the CLarification Request (CLR) answers and input specification 
updates as well as change requests. Once the SRS document has been released, the CLR shall 
be used to ask question or request or give clarification on SRS (the CLR is used with the 
carmaker and internally in the team). The SRS shall be updated with the content of the CLR. 
The change of the specification after the specification freeze milestone shall be an exception. 
The specification freeze corresponds to the date where in theory no specification change is 
allowed. 
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<Variant 1> Level description 
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Figure 2.7 - A UML:like data model of the SRS currently used in Johnson Controls 
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Diagnosis 5 


The Software Requirement Specification (SRS) document is often a large document (about 
hundreds of pages), difficult to manage, incomplete and not regularly updated. 


Diagnosis 6 


Sometimes, the SRS document is the official and contractual document between carmaker 
and supplier. It is also the main document used by the development and V&V teams in their 
activities. It has a standard structure but there are no standards to specify carmakers’ 
requirements (more especially functional requirements). 


F. Quality criteria of a requirement 


In order to support reviews and improve the quality of software requirements the following 
criteria are defined by Johnson Controls software experts. They must be checked during 
reviews and respected during the set up of the SRS document. The review of the SRS 
document is supported by a Checklist. It consists of verifying explicitly the criteria listed 
hereafter. 


1. General criteria 


e Use a simple language style to define the requirements. 

e Short sentences, not interlocked, need. 

e Use present tense. 

e Use simple, clear, vocabularies, introduced terms wherever possible 

° No multiple definitions. 

e Reference what is defined correctly by existing specifications (do not copy). 

e _ Apply SRS template and Requirement Management tool template (Styles, fonts, types 
cd: 

° Do not specify design or implementation. Describe what to do not how to do it. 

e In case of complicated conditions use state-, sequence- or flow chart to gain clarity and 
remove ambiguity. 

e  Describe the interface of the device with the environment not the interface of 
components Within the device. In case of complicated conditions use state-, sequence- 
or flow chart to gain clarity. 


2, Detailed Quality Criteria 


Understandable 


e The text is easy to understand and the requirement 1s clear for the reader. 
e _ Needless or confusing words are not used. 


Complete 


e The Feature (group of requirements) will contain all the information needed to 
implement and test the requirement. No information needed to implement the feature 
will be missing. 
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Consistent 


e There will be no contradictions within a single requirement or between two 
requirements. 
e Usage of the same terms as used in other definitions. 


Necessary 


e The definition is needed for the realization of the feature. Removing the requirement 
will change the behavior related to the feature and/or will render the feature 
incomplete. 

°  Unambiguous. 

e The definition 1s clear and has only one single interpretation. 


Atomic 


° A further breakdown of the definition is not possible. 
Feasible 

e _Itis possible to implement, fulfill and test the definition (time, budget, know-how?). 
Maintainable 


e There will be no redundancy. Redundancy is allowed only when removing the 
redundant phrase, sentence or requirement will cause ambiguity. 


Testable 


e  Itis possible to test the definition / to develop a test case. 
V. Software verification and validation activities in automotive industry 


A. Overview on software verification and validation techniques at Johnson 
Controls 


As we shown in Section 3, Within each step of the software standard life cycle, engineering 
activities are performed in an iterative way according to the standard V-model of the software 
industry. The Component Development process is the process where the source code is 
developed. Following the code implementation and before any carmaker delivery, a series of 
verification and validation techniques have to be applied on the source code in order to check 
its correctness and its compliance with the carmaker expectations (software requirement 
specification). At Johnson Controls, we identify 3 software inspection techniques (Code 
review, static analysis, dynamic analysis) and 3 software test techniques (unit, integration and 
validation test). In this dissertation, we adopt the definitions proposed by ZEEE for each of 
these techniques. 


Definition 2.7: Software code review (IEEE Std. 610 1990) 


The software code review is a visual examination of a software work product to detect defects, 
e.g. violations of development standards and non-conformance to higher level documentation. 
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Definition 2.8: Static code analysis (LEEE Std. 610 1990) 


The static code analysis is an analysis of source code without execution of that software. À 
static code analyzer is tool that carries out static code analysis. The tool checks source code, 
for certain properties such as conformance to coding standards, quality metrics or data flow 
anomalies. 


Definition 2.9: Dynamic code analysis (LEEE Std. 610 1990) 


The dynamic code analysis is the process of evaluating behavior, e.g. memory performance, 
CPU usage, of a system or component during execution. The dynamic analysis tool provides 
run-time information on the state of the software code. These tools are most commonly used to 
identify unassigned pointers, check pointer arithmetic and to monitor the allocation, use and 
de-allocation of memory and to flag memory leaks. 


Definition 2.10: Software Unit, Integration and Validation test (IEEE Std. 610 1990) 
Unit test is the test of individual software components. 


Integration test is the test performed to expose defects in the interfaces and interaction 
between integrated components. 


Validation test is the process of testing an integrated software product to verify that it meets 
specified requirements. 


The interactions between these testing techniques are illustrated in Figure 2.6. 


Unittested 


CMP 1 
components | 
Integrated Validated 
: . software 
He product 
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test test ———————> 
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Figure 2.8 — Interactions between unit, integration and validation tests 


The location of these techniques within the Johnson Controls software process map is shown 
in Figure 2.9. 
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Code Static Dynamic Unittest Integration Validation 
review analysis analysis test test 


Figure 2.9 — Software verification and validation techniques within the Johnson Controls 
process map 


Each of these techniques must catch different classes of bugs at different points in the 
development cycle. 


B. Software V&V techniques in Component Development process 


The purpose of the Component Development process, as it is defined in Johnson Controls, 1s 
to produce executable software components that properly reflect the global design and 
software requirements. Moreover, a strategy has been defined in order to verify and validate 
each software component, after it is produced. This strategy is applicable for all the software 
components in the projects. For each component in the scope of the V-model, a Component 
Development process is used. After analyzing internal documents related to the definition of 
the Component Development process, we identify 3 main activities when developing a new 
software component (Cf. Table 2.6). 
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Devélop detailed: Bugs detected on component can lead to 
design correct detailed design and/or code 


Produce 
component 


The "Verify component" activity consists to: 
Verify : 
component - Review the code 
(e] 


-  Analyze statically the code 


N -  Analyze dynamically the code 


“Unit test” of the software component 


If the results of the unit test done on the 

component are OK, the component is 

promoted to integration 

Allthese activities are performed by Software Developers (SD) 
Table 2.6 — Process flow of the Component Development process 


First activity: Produce component 


Produce component activity aims to produce components (source code, code generator data 
.…) and/or to fix bug(s) detected in next steps of development (verification and unit test 
activities). This coding activity is based on global and detailed design and implements the 
component design made previously. Coding has to be done in accordance with defined coding 
standards, rules and guidelines and with embedded system constraints (memory size, 
hardware dependency ..….). 


Second activity: Verify component 


In this activity, three software inspection techniques (static V&V techniques) are performed: 


Code review, based on the Review & Verification process. 
e _ Static analysis based on a commercial tool (QAC°). 


° Dynamic analysis based on a commercial tool (PolySpace”?). 


The verify component activity follows a process flow described in Table 2.7. 


*lhttp:/www.programmingresearch.com/QAC_MAIN.html (Consulted on November 2008). 
7 http:/www.mathworks.com/products/polyspace/index.html (Consulted on November 2008). 
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Process Comments 


Produce 
component 


Do we need to review the software 
component ? 


Code review is performed in filling 
the nonconformities in an /ssue Log. 


Do we need to statically analyze the 
software component? 


Static analysis is performed with a 
static analysis tool: QAC. 


Do we need to dynamically analyze 
the software component? 


Dynamic analysis is performed with 
a dynamic analysis tool: PolySpace. 
It is possible to perform dynamic 
analysis only for the whole software 


Table 2.7 — Process flow of the verification of a software component 


Code reviews are mainly intended for checking respect of coding standard and rules / quality 
of comments in a software component. Sfatic analysis 1s intended to check the compliance of 
the source code with the international automotive software coding rules (MISRA-C”). 
Dynamic analysis 1s intended for detecting problems, with a dynamic point of view, early in 
the life cycle. This type of problems could be detected by testing activities. Static and 
dynamic analysis can be done on the whole software (and not only for each component). 


Third activity: Unit test component 


This activity consists of testing unitarily each software component. In other words, this 
software test technique intends to verify the correctness of all functions / conditions / 
decisions / component inputs and outputs / boundaries and limits in a component source code. 


7 MISRA:C is a software development standard for the C programming language developed by MISRA (Motor 
Industry Software Reliability Association, http://www.misra.org.uk/, Consulted on November 2008). 
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To summarize, the software Component Development process within Johnson Controls 
performs four V&V techniques on each software component: three inspection techniques 
(code review, static and dynamic analysis) and one test technique (unit test). In the 
following, we develop each of these techniques as it is practiced at Johnson Controls. 


d: Technique 1: Review of a software code 


According to the Johnson Controls process, the purpose of the software component review 1s 
to: 


Check whether the source code of the produced component respects the design or not. 
e Check whether all remaining problems after automatic static/dynamic analysis are 
properly justified in the source code. 
Verify the quality of the comments written in the source code. 
e  Verify the traceability of the component to software requirements. 
Check whether the source code respects the coding guidelines, especially those rules 
that cannot be tested automatically by a tool. Software engineers have to check the 
compliance of the inspected code to the rules and recommendations. The 
modifications of these rules can be made by a committee, whose members are 
appointed by the Software Engineering Process Group°* (SEPG) of the company. The 
committee includes representatives of all Johnson Controls sites on which this 
document is deployed. In fact, there is a document which defines coding rules and 
recommendations for using the C language”” in the development of embedded 
software for the automotive industry. The document is organized as a collection of 
rules and recommendations illustrated in Figure 2.10: 
© À rule is a prescription that has mandatory character. It must be always 
followed. 
o À recommendation is a prescription that has advisory character. It must be 
followed as much as possible. 


General rules 


Code structure 
Comments 


Code layout 
Source code and files ICS “EYON 


| File inclusion 
Coding rules and Naming rules 


recommendations 
Functions 


Flow control 
Type 


RE Constants 
Rp Fret Structures and Unions 


Variables 
Arrays and Pointers 


Figure 2.10 — Classification of programming rules and recommendations 


7 The SEPG is a group of software experts. 
Computer language. 
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An excerpt of these rules and recommendations is illustrated in Appendix A. Absolute and 
unconditional adherence to these rules and recommendations may not be possible at all times. 
However, it must be noted that some general rules is of an outstanding importance — under no 
circumstances shall this rule be broken. 


In order to perform the code review of a software component, a group of Johnson Controls 
software engineers must read the source code and simultaneously fill an Zssue Log With the 
identified issues. An /ssue Log capitalizes the reviewers” names, the date of reviewing, the 
name of the reviewed component, the review load, the reviewed number of Lines Of Code 
(LOC) and a list of identified issues. For each issue, reviewers give an /D, the number of the 
line where the issue was found, a description of the issue and the status of the decision and 
correction. 


In 2006, we carried out a study within Johnson Controls on four projects related to two 
different electronic products. The aim of this study was to audit the practical implementation 
of the code review technique. The main result of this study 1s that number of code reviewers 1s 
not aware of the coding rules and recommendations. Moreover, we note that code review 1s 
not systematically performed on each new software component. These conclusions were also 
validated by inner software experts and managers. In fact, in automotive industry, software 
testing is considered to be the main V&V activity which has to detect all the bugs. 
Unfortunately and as shown in Figure 1.11, detecting bugs later in the process costs more 
than detecting them as soon as they are introduced. 


Diagnosis 7 


Sometimes, the review of software code is badly done or even ignored. In fact, number of 
code reviewers is not aware of the coding rules and recommendations to be checked. 
Moreover, the code review is not systematically performed on each new software 
component. In consequence, the code review does not often detect all the bugs that must be 
detected through this activity. 


2. Technique 2: Automatic static analysis of a software code 


According to the Johnson Controls process, the goals of the static analysis are to: 


Improve the quality of the source code. 

Improve the robustness of the software. 

Make the source code as much as possible portable. 
Be compliant with MISRA-C. 


There are two phases of this analysis, executed separately: 


e During the verification activity of a single component. This must be done by the 
developers who create/modify the components. 

e Overall project static analysis. Done after the integration of all the components. This 
task could be delegated to an experienced developer. 


The static analysis is performed automatically using a computer tool such as QAC, the most 
used in automotive industry. It is recommended to apply this V&V technique in the beginning 
of the project in order to be able to detect and fix the issues as early as possible. The criterion 
to stop the sfatic analysis of a source code is that all QAC errors and warnings are either fixed 
or justified. À screenshot of the QAC tool is illustrated in Appendix À. 


3. Technique 3: Automatic dynamic analysis of a software code 
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À dynamic analysis of the software code 1s performed in order to find and fix as early as 
possible the software bugs that could possibly occur one executing the software product and 
cannot be detected by code reviews and static analysis. The dynamic analysis is performed 
automatically thanks to a commercial tool PolySpace. The intention 1s to get a clear view 
about the dynamic behavior of the software early in the process. The earlier detection of 
problems reduces the risk of having serious bugs at late project phases. However, the project 
has to plan enough time (depending on the project’s size and warnings reported) for warning 
analysis. The criterion to stop the dynamic analysis of a source code is that all Polyspace 
errors and warnings are fixed or justified. À screenshot of the Polyspace tool is illustrated in 
Appendix À. 


4. Technique 4: Unit test of a software component 


The unit test of a software component is described by the process flow of Table 2.8. After 
analyzing many Johnson Controls documents related to the unit test activity and interviewing 
inner engineers practicing unit test, We identify three main activities: design the fesf cases, 
review the test cases and finally execute the rest cases and analyze the results. We illustrate 
below the definition of a fest case as it is adopted in Johnson Controls. 


Definition 2.11: Test Case, Test Step and Operation (Johnson Controls) 


Let us consider a functionality with two input signals: 11(with domain D(11)={0,1}) and 12 
(D(L2)={1,2,3}) and three output signals: OI(D(01)={0,7,14}), O2 (D(O2)={1,2,3}) and O3 
(D(03)={0,1}). We first call “Operation”, the fact that an input signal is set to a value. For 
example, 12=3 is an operation. À “Test Step” is composed from an operation, an inter- 
operation time and expected results on the output signals. A “Test Case” is a succession of 
“test steps”. 


An excerpt from a test case designed is given in the Figure 2.11: 


- In test step 9,6, test engineers wait for 500 ms without carrying out any actions on the 
product and check that the outputs of the product haven't changed. 


- In test step 97, test engineers activate a switch (Input_1=1), wait for 200 ms and check that 
the concerned outputs are activated according to the expected behavior. 


Test Step No Test Actions Expected Results 


Output_1=0 
Output_2=0 
Output_3=0 


Test# 96 
Wait 500 ms 


Expected results 
on output signals 


Figure 2.11 — An excerpt from a test case (two test steps) as designed by Johnson 
Controls tester engineers 
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Process Comments 


Unit test co 
Test cases for unit test are manually 


designed. The quality of these test cases 
1s based on the experience of the 


developer. 


Review Designed test cases are reviewed 
test cases 


Designed test cases are executed on the 
software component under test 


If the results of the unit test done on the 
Execute test cases component are OK, the component is 
& analyze results : ù 

promoted to integration 


Table 2.8 — Process flow of the unit test of a software component 


After developing a software component, software developers design manually test cases for 
the unit test of this component. The main purpose of the unit test is to cover at 100% the 
source code. It is the main criterion to stop testing unitarily a software component. The 
principles of code coverage are developed in Section 6.A. In fact, developers analyze the 
structure of the software component and design test cases that must cover all the source code 
of the component under test. The fest case design process presently used by the engineers at 
Johnson Controls is deeply described in Section 6. It is important to note that a software 
component 1s about 2000 LOC (without blanks and comments), a reasonable number of LOC 
to be analyzed (according to experts). The technique of designing fest cases while having 
access to the code of the software under test is called structural or white-box or program- 
based test. À survey on software testing techniques (Bernot 1991, Beizer 1995) is provided in 
Chapter 4 — Section 3.B. 


Presently, in Johnson Controls, the unit test is not responsible to verify the compliance of a 
software component with the carmaker requirements. In fact, one software component can be 
tested unitarily (100% of code coverage) Without fulfilling the behavior required by the 
carmaker. 
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Diagnosis 8 


According to the “To Be” process, the unit test of a software component must ensure a 
100% source code coverage. However, this V&V technique is not responsible to verify the 
compliance of the component’s behavior with the carmaker requirements. One software 
component can be tested unitarily (100% of code coverage) without fulfilling the behavior 
required by the customer. 


The language used to design test cases for the unit test of a software component is the C 
language. À standard unit test structure providing predefined C functions in order to help the 
test engineer writing fest cases 1s developed in Appendix B. 


The designed test cases are reviewed in order to check: 


the relevance of designed fest cases in regard with the tests objectives, 

e the reached code coverage, 

° and the usefulness of the dummy test cases dedicated only to reach the expected code 
coverage. 


Finally, all the designed test cases are executed on the software component under test. AI the 
dependencies and connections between the components are simulated to isolate the tested 
component from the project. Test results are analyzed to decide 1f Component Development 
activities have to be restarted, in case of failed tests. The code coverage is recorded and used 
as criteria to stop the design of test cases. The unit test execution platform is developed in 
Appendix C. 


C. Software verification and validation techniques in Integration process 


Once a set of components are produced and verified through the Component Development 
process, they are integrated together and an integration test (V&V technique) is performed on 
the overall software product. In Johnson Controls, the purpose of the /ntegration process is to 
assemble the software product from the software components, ensure that the software 
product, as integrated, functions properly, and deliver the tested software product to 
Validation process. 


1. Technique 5: Integration test of a set of software components 


The purposes of the integration test are to ensure that global design requirements work as 
expected and the quality of the software allows the execution of the Validation process. To do 
this, three steps have been defined by Johnson Controls software experts: 


First step: Interface review 


The engineer responsible of integrating the software components reviews each component in 
order to verify the conformity of interfaces to predefined architecture. According to the 
relevance of the review, she/he can decide to setup additional verification by adding test steps 
in the functional and/or change test. 


Second step: Change test 


Change test cases are defined and executed only once, when change 1s integrated. The first 
objective of change test is to verify the good implementation of the requirements involved in 
the change. The second objective of change test is to verify the good implementation of the 
architecture expectations involved in the change. Change can be either evolution 
implementation or bug fixing. For bug fixing, change test cases have to check that bug is not 
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reproduced when sequence used for bug detection 1s re-executed. For evolution, change test 
cases have to check main impacts of the change on software requirements. 


Third step: Functional test 


Functional test cases improve confidence on the verification. The purpose of this test is to 
detect regression on a new integration. It has to check a limited list of software requirements. 
In this perspective, a set of fest cases per software functionality has to be defined. 


In conclusion, the criteria to stop the integration test of a software product are rather 
subjective. Indeed, the engineer must verify (according to her/his point of view) that a change 
does not impact the whole software product and that the main requirements of each 
functionality are satisfied. 


D. Software verification and validation techniques in Validation process 


Once a set of components are integrated together, a validation test (V&V technique) is 
performed on the overall software product. 


1. Technique 6: Validation test of a set of software components 


In Johnson Controls, the purpose of the validation test is to confirm that the integrated 
software meets the carmaker requirements related to software. The validation test is activated 
at each new iteration when the software product has been successfully integrated. Now, the 
validation test is completed when: 


e The planned validation procedure is executed. À validation procedure is composed 
from a set of fest cases. 
AI of the requirements (defined in the SRS) in the scope of the delivery are covered. 

e And if any testable requirement could not be covered, the reason about it must be 
justified. The requirement coverage, as it is currently practiced in Johnson Controls, 1s 
developed in Section 6.B. 


For each iteration, the validation test of a software product 1s described by the process flow of 
Table 2.9. After analyzing many Johnson Controls documents related to the validation test 
activity and interviewing inner engineers practicing validation test, We identify two main 
stages: Preparation of Validation and Execution of Validation. 
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Process Comments 


lteration start 


The strategy for Software Validation 
Develop software : ; 
validation plan is defined 


Désionaalidatio Test cases are identified 


procedure 


Implement validation Test cases are designed 
procedure 
EE ————— 5) 


= T | 

- Is there any new software 
integration? 
YES ë ‘ 


Re Execute the selected test cases on 
the new software integration 


ZO—12>VD>UMAIT 


YES 
ie Is it the final software product? 
YES 


Perform full Execute the whole fest cases on the 
VAlRAION final software product. 


ZO—-COmMmXxXm 


Iteration stop 


Table 2.9 — Process flow of the validation test of a software product 


a. Preparation of validation 


Develop the software validation plan 


The Software Validation Plan (SVP) describes the validation strategy for a project. It serves 
as a guideline for executing validation tasks by project and by scope of delivery. The strategy 
of validation may be adjusted for each iteration (according to the delivery content). The SVP 
supports the following objectives: 


1. Define the validation test execution platform, the necessary equipment and the common 
and reused validation components. A detailed description of the validation test execution 
platform is performed in Appendix C. 

2. Recommend and describe the strategy for validation test application. The SVP indicates, 
for each software functionality in the scope of the delivery, the rypes of validation tests to 
be performed and if the execution of the corresponding tests is manual or automatic. In 
Table 2.10, a description of the test types is provided. 
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Type of test 


Functional Ensure the proper functionality of the whole software 
Data integrity Ensure the proper functionality of the whole software during Data 
treatment 


Failover and recovery Ensure the proper functionality of the whole software during 
restoration process 


Configuration Ensure the proper functionality of the whole software when different 
system configurations are set 

User interface Ensure the user's interaction with the software 

Performance Ensure time-sensitive requirements: response times, transaction 
rates, etc. 

Load Ensure time-sensitive requirements: response times, transaction 
rates, etc. 

Stress Ensure time-sensitive requirements: response times, transaction 
rates, etc. 

Long term Ensure time-sensitive requirements over a long period of time 

Security and data Ensure the proper accessto specific functionalities 

access control 

Installation Ensure the proper softwareinstallation process 


Table 2.10 — Description of the type of tests used in validation test at Johnson Controls 
(Apostolov 2007) 


3. Establish the regression strategy. Two kinds of regression strategy are defined: change 
oriented and priority oriented. The purpose of the change oriented strategy is to define 
how to test a software product after new functionalities or changes are implemented / 
applied, in order to ensure that the implementation of not changed requirements is not 
impacted by the changes. The aim of the priority oriented strategy is to ensure that the 
quality of implementation of requirements having highest priority has not regressed while 
adding new capabilities. 


Design the validation procedure 


The purpose of this activity 1s to identify the structure of the whole validation procedure and 
to establish a link between fest cases and requirements. In other words, this activity aims to 
identify the number of required fest cases and describe the scope of each one. The software 
requirements (from the SRS) defined as scope of the following delivery and the SVP are used 
as inputs for the design of the validation procedure. In Johnson Controls, a list of good 
practices when designing the validation procedure has been established: 


e  Itis recommended that, for each software requirement, at least one test case has to be 
defined and one fest case could cover more than one requirement. 
One test case can cover only one type of test. 

e  Atest case must cover all the aspects and combinations of a requirement. 


Implement the validation procedure 


The aim of this activity 1s to design, in a step by step manner, the test cases of the validation 
procedure. Based on each test case scope, validators analyze the carmaker requirements and 
design the test case that must verify the compliance of the software product with the 
corresponding requirement. The test case design process presently used by the engineers at 
Johnson Controls is deeply described in Section 6. It is important to note that validators do 
not have access to the source code of the software product under test. It is a considered as a 
black-box with a set of inputs and outputs. The technique of designing test cases Without 
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having access to the source code of the software under test is called functional or black-box or 
specification-based test. À survey on software testing techniques (Bernot 1991, Beizer 1995) 
1s provided in Chapter 4 — Section 3.B. 


The language used to design test cases for the validation test of a software product depends on 
the validation test execution platform. ]n case of an automatic execution of the test cases, one 
uses a script language. It is a Johnson Controls property language very similar to the well- 
known Visual Basic”° language. A detailed description of this language is provided in 
Appendix B. In case of a manual execution of the fest cases, test cases are written in natural 
language. 


b. Execution of validation 


The validation is executed in the following sequence: 


1. Configuration and initialization of the validation test execution platform. 
2. Execution of the fest cases in sequence defined by the regression strategy defined in 
the SVP (incremental or full validation). 


During the execution of the rest cases, “OK” and “NOK” results, which are prepared by 
observing and comparing the expected and the observed results, are set for each test step. The 
execution of the validation procedure could performed either automatically with the help of a 
tool or manually (Cf. Appendix C on the validation test execution platform). I…n the case of a 
“NOK” result, the comment describing the observed situation must be added and a bug has to 
be issued in the problems’ tracking tool. À detailed description of the Johnson Controls 
problems” tracking tool is performed in Section 7. 


VI. The test case design process presently used in automotive industry 


Our audit (Cf. Section 5) on the software V&V activities within Johnson Controls has 
confirmed the proposal of the National Institute of Standards and Technology (Cf. Definition 
1.5): “Software testing is a Widespread V&V technique in automotive industry”. In fact, we 
notice that each of the Development (unit test) and Validation (validation test) processes 
perform software testing in order to verify and validate the correctness of the software 
delivered at the end of the process. 


Presently, most of automotive suppliers have a manual test design process. As the software 
products become more and more complex (Cf. Chapter 1), it is 1llusory to be able to check 
that the software product responds correctly to all possible operations. In Chapter 8 — Section 
2, we further demonstrate that software testing is a NP-Complete problem and therefore it 1s 
impossible to be able to cover all the operation space. In fact, for each software component or 
product under test, we can associate a potential operation space (Cf. Figure 2.12). Each 
engineer has a different perception of the possible and critical operations (based on her/his 
experience). Therefore, based on a common fest objective, two engineers could choose 
different test cases according to their perception. In Johnson Controls, a software component 
or product is always tested against predefined objectives. 


? Computer language (http://msdn.microsoft.com/en-us/library/sh9ywfdk(vs.80).aspx, Consulted on November 
2008). 
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Engineers vision of 
the operation space 


Æ Software 
12 — | component 
! or product 


Vi 
Output 


Operation 


Discrete Domains 


> 


Potential operation space 
Figure 2.12 — Potential operation space of a software 


A. Design test cases for the unit test 


In Johnson Controls, the main purpose when testing unitarily a software component is to 
cover at 100% the source code of the component under test. Developers analyze the structure 
of the software component under test (White-box test), select one operation Within the 
potential operation space and choose a time to wait before a next operation (Cf. Figure 2.13). 
Afterwards, by analyzing the source code of the component, they assess the expected values 
to be checked on some output signals. We note that developers do not check the behavior of 
all the output signals of a software after each operation. In fact developers decide to check 
only some output signals in relation with the performed operation. In fact, they verify the 
expected behavior according to their understanding of the program behavior. If the designed 
test steps allow to cover all the source code, developers stop designing test steps. If not, 
developers analyze deeply the uncovered pieces of code with the goal of designing one or 
more fest steps that cover these pieces of code. Sometimes, for time and budget reasons, 
managers could decide to stop testing unitary a software component even if the 100% code 
coverage 1s not reached. The principles of code coverage are developed in the next section. 


What are the expected 
results on the outp. 
signals? 


Obiective: Cover at 100% 
the source code ofthe 
componentundertes 


100% code coverage 
Time and Budget 


Operation and Inter- 
operationtime À Human 


à Operation EE 
selection based on Developers ne source code … bevelop 


the engineers (. l'operation time analysis 
experience 3 


Expected Tes 
results on 


Software 
component 


Discrete 
Domains 


| Human SUN 
source code Developers 
analysis 


Figure 2.13 — Johnson Controls present approach to design a test case for unit test 


Quality of the design of test cases for automotive software: design platform and testing process 


93 


Industrial audit KR. AWEDIKIAN 


1. Code (structural) coverage 


A survey on code coverage based testing tools is done in (Yang 2006). In Johnson Controls, 
the code coverage is measured by a commercial tool (C -Cover””). Code coverage is a way to 
measure how thoroughly a set of fest cases covers a code (Cf. Figure 2.14): 


+ The coverage rate of statements: This metric reports whether each executable line of 
code is encountered. 

e The coverage rate of procedures: This metric reports whether the fest case invokes 
each procedure (or function) of the software. It is useful during preliminary testing to 
assure at least some coverage in all areas of the software. 

+ The coverage rate of decisions: This metric reports whether boolean expressions 
tested in control structures (such as the if-statement and while-statement) evaluated to 
both true and false. The entire boolean expression is considered one frue-or-false 
predicate regardless of whether it contains logical-and or logical-or operators. 
Additionally, this metric includes coverage of switch-statement cases, exception 
handlers, and interrupt handlers. 

+ The coverage rate of conditions: Condition coverage reports the frue or false outcome 
of each boolean sub-expression, separated by logical-and and logical-or if they occur. 
Condition coverage measures the sub-expressions independently of each other. This 
metric is similar to decision coverage but has better sensitivity to the control flow. 
However, full condition coverage does not guarantee full decision coverage. 


For instance, the piece of code of the Figure 2.14 has: 1 procedure, 1 condition, 2 decisions 
and 8 statements. 


Procedure AnswerYesNo() CONDITION 
LU | Var = « » Met 
Si Whilevar<> « Yes » ou 
A: Writ 
LL : 
Œ End While N, DECISION 
[a Send var STATEMENT 

: End Procedure (Line of Code) 


Figure 2.14 — Code (structural) coverage indicators 


These criteria are apparently relevant since the goal of the festing activity is to check if all the 
pieces of the software have been visited. But it is not that simple (according to experts)! 


In 2006, we analyze the unitary test reports on more than 5 projects related to different type of 
products. We also discuss these reports with inner software experts. In fact, even if the 100% 
code coverage is not reached, managers can decide to stop testing unitary each software 
component for time and budget reasons. 


Diagnosis 9 


Sometimes, the unit test of a software component is incomplete or even inexistent. In other 
words, the source code of the component under test is not covered at 100%. As a 
consequence, the uncovered pieces of code could hide critical bugs. 


7 http:/www.bullseye.com/productInfo.html (Consulted on November 2008). 
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B. Design test cases for the validation test 


Presently, in Johnson Controls, the unit test is not responsible to verify the compliance of a 
software component with the carmaker requirements. In fact, once a set of unitarily tested 
components are integrated together, validators have the responsibility of verifying the 
compliance of the whole software product with the carmaker requirements. To do this, 
validators analyze one or more software requirements (Black-box test) and select one 
operation Within the potential operation space (Cf. Figure 2.15). Afterwards, by analyzing 
the carmaker requirements, they assess the expected values to be checked on some output 
signals. Idem to the design of a test case for the unit test, validators decide to check only 
some output signals in relation with the performed operation. In fact, they verify the expected 
behavior according to their understanding of the carmaker requirements. If the designed test 
steps allow to cover the carmaker requirements under test, validators stop designing test 
steps. If not, validators analyze deeply the considered requirements with the goal of designing 
one or more fest steps that cover at 100% the requirements under test. Sometimes, for fime 
and budget reasons, managers could decide to stop validating a software product even if the 
100% reguirement coverage is not reached. However, the carmaker must be notified on the 
uncovered requirements. The requirement coverage, as it is currently practiced in Johnson 
Controls, is developed in the next section. 


Obiective: Cover at 100% 
oneor more carmaker. 
software requirements 


What are the expected 
results on the outp 
signals? 


Oo, 100% requirement coverage 


RUE (e) 
SA : Operation and Inter- Human ° ose Time and Budget Test Case 
sl do operationtime & A carmaker RUN | = - 
à 3 9 selection based on Validato peuten ft . Tesuts on 
ê | É 9 : : “ Inter- SOjtWare Developers | some output 
! = 2) L : i ï 
Q à the engineers /_.) loperation time| "euirements signals 
min ë lex À 
experience mars, analysis NOK 


Discrete 
Domains 


Which pieces ofthe 
consideredrequiremen 
are not covered: 


Human 
carmaker FA 
Le à 


software Developers 
requirements 
analysis 


Figure 2.15 — Johnson Controls present approach to design a test case for validation test 


In 2007, we analyze the bugs of two different projects related to two different electronic 
products. It is important to note that, in Johnson Controls, bugs detected during review and 
unit test activities are often not capitalized in the problems’ database (Cf. Section 7). Once a 
bug 1s detected during these activities, it 1s corrected immediately by the person who detects 
it. Therefore, most of the capitalized bugs are detected in validation test. Through our study, 
we note that up to 30% of the stored bugs are due to errors in the design of the rest cases in 
validation test. In fact, validators do not assess correctly the expected values to be checked on 
the output signals. This could be explained by the fact that a human assessment of a program 
behavior could be inaccuracy since carmaker requirements related to the software domain 
become more and more complex. 
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Diagnosis 10 


When testing a software component or product and after an operation on the input signals, 
test engineers do not check the behavior of all the output signals of the component or 
product under test. Based on their understanding of the program behavior and/or the 

carmaker requirements, test engineers decide to check only some output signals in relation 

with the performed operation. In fact, they verify the explicit expected behavior but not the 
implicit one. 


1. Requirement (functional) coverage 


The criterion of code coverage does not directly assess the compliance of the software 
component or product with the carmaker requirements; this is a biased indicator. In fact, the 
requirement coverage is related to the coverage of the functional requirements of the software 
under test. Through a literature review (Dalal 1988, Bontron 2005, Yang 2006), several stop 
testing criteria based on covering software requirements have been identified in Chapter 4 — 
Section 4.B.1. They primarily deal with the fransitions coverage of a graph-based 
specification. At Johnson Controls, the carmaker requirements related to the software domain 
are referenced and managed using the SRS model and the coverage rate of these requirements 
is mainly used as the criterion to stop validation test. Moreover, the requirement coverage 1s 
measured subjectively by the validator. Paradoxically and even a 100% coverage of the 
functional requirements has been reached during the festing of a software product, the 
carmaker is able to detect a nonconformity between the code and their requirements. In fact, 
presently, one requirement can hide two or more other requirements. Let us consider in Figure 
2.16 an excerpt of software functional requirements as they were defined by a Johnson 
Controls engineer. These requirements have two inputs and one output: 27 (with domain 
D(1])={0,1}), 2 (D(2)={0,1}), O7 (D(O1{0,1}). 


Requirement 1: 
In case of input /1 is equal to 1 and input /2is 


equal to 0, therefore the output O2 must be set to 0 


Requirement 2: 
In other cases, Output O2is always set to 1 


Figure 2.16 — An excerpt of software functional requirements as defined by a Johnson 
Controls engineer 


During the validation test, one inexperienced validator designs one fest case (composed from 
two test steps) in order to cover the previous requirements: 


- Test step 1: set I] to 1, 22 to O and check if O] is equal to O 
- Test step 2: set 11, 12 to 1 and check if O] is equal to 1 


Therefore, she/he decides to stop testing these requirements and to set them as covered. In 
fact, through test step #1, the validator covers at 100% the first requirement but rest step #2 
does not cover at 100% the second requirement. Indeed, the second requirement can be split 
into three “implicit” requirements to be tested: 


° In case of input Z] is equal to 1 and input 22 is equal to 1, therefore the output O2 must 
be set to 1 — covered by test step #2 

° In case of input Z]J is equal to 0 and input 2 is equal to 1, therefore the output O2 must 
be set to 1 — not covered by the test case 

° In case of input Z]J is equal to 0 and input 2 is equal to 0, therefore the output O2 must 
be set to 1 — not covered by the test case 
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This could lead to the conclusion that the present Johnson Controls definition of a requirement 
is not enough refined. 


Diagnosis 11 


The present definition of a software requirement is not enough refined. In fact, one 
requirement can hide two or more implicit requirements. Therefore, inexperienced 
validators could miss testing some of the carmaker implicit requirements. 


Based on our analysis of the present Johnson Controls approaches to design fest cases for unit 
and validation test, we do the four diagnoses listed below. 


Diagnosis 12 


In validation test and after selecting an operation to be performed on the software under 
test, test engineers analyze the carmaker requirements in order to assess the expected values 
to be checked on some output signals of the software. In fact, this assessment is based on 
the engineers’ understanding of the requirements and may lead to errors. 


Diagnosis 13 


For each software component or product under test, a large potential operation space is 
associated. Each engineer has a different perception of the possible and critical operations 
based on her/his experience. Therefore, the present strategy to select operations in order to 

test a software is irrelevant. 


Diagnosis 14 


The test cases designed by engineers do not always simulate the real use of the software 
product under test. The main purpose of testing activities is to cover the software code and 
requirements. As a direct consequence, basic user operations on the product could be not 
tested by the supplier before a carmaker delivery. 


Diagnosis 15 


Presently, the test cases for a software are manually designed by engineers. As the size of 
automotive software growth, this task becomes a laborious task and accounts for more than 
50% of the total time and budget of a project. 


VIL. Capitalizing bugs in Johnson Controls 


A. Snapshot on the Johnson Controls problems” tracking tool 


Johnson Controls as many other electronic suppliers uses a problems’ tracking tool 
(TeamTrack”*) in order to manage and store problems detected during a project. À snapshot of 
this tool is 1llustrated in Figure 2.17. 


# http:/www.serena.com/products/teamtrack/index.html (Consulted on November 2008). 
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Figure 2.17 — Screenshot of the problems’ tracking tool 


Problems are classified according to four categories: software, hardware, mechanical and 
others. In the following, we focus on software problems, called bugs. The fracking tool has a 
database where all the problems are stored by project. In fact, a project is the combination of a 
customer (for instance, Renault), a type of product (for instance, a body controller module) 
and a car platform (for instance, Laguna platform). The problems’ database has been created 
in the late 90’s and now we estimate to tens of thousands the number of capitalized software 
bugs. According to experts, about 60% of these bugs are “true” bugs. The remaining 40% are 
duplications or without continuation. Moreover, in 2006, we perform a study on the 
capitalized software bugs and we come up to the conclusion that up to 90% of these bugs 
were detected during the validation test activity. Software experts and managers confirm that 
the bugs detected during the other V&V activities (review and unit test) are often not 
capitalized in the problems’ database. Once à bug is detected during these activities, it 1s 
corrected immediately by the person who detects it. Most of the capitalized software bugs are 
detected in validation test. 


B. The bug’s model currently used in Johnson Controls 


One of the support processes (Cf. Figure 2.2) has the responsibility of ensuring that all found 
software bugs and all changes on the product are identified, analyzed, managed and controlled 
to resolution and implementation. In fact, once an engineer has recorded a bug in the 
problems’ database, a workflow process is initiated between the team members in order to: 


assess the impacts of the bug, 

make decisions, 

plan and implement the corrections, 

and finally verify and validate the non-regression of the software product. 


Apart the evolution of the bug status (dynamic view) since its creation and till its resolution, 
we focus on the bug’s model (static view) currently used in Johnson Controls. Fifteen years 
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ago, Sagem” software experts developed a bug’s model with the aims of 1) managing the life 
cycle of a problem, 2) having traceability of the problems detected internally and by the 
carmaker, 3) monitoring a project by the number of detected, corrected and uncorrected 
problems and finally 4) reusing (by experts analysis) critical stored problems to avoid similar 
problems on future developments. In fact, a total of 111 attributes should be filled in by the 
engineers for each capitalized problem. We analyze about 2000 bugs from two different 
projects and products and we come up to the conclusion that 75% of these attributes are filled 
in; the remaining 25% are systematically unfilled. On the 75% filled attributes, 25% of these 
attributes are free fields. In Figure 2.18, we classify the 111 attributes according to the major 
aspects of a software bug (Mellor 1992, Fenton 1996): location (9 attributes), timing (28 
attributes), symptom (2 attributes), impact (2 attributes), cause (7 attributes), type (14 
attributes), severity (7 attributes) and cost (2 attributes). The remaining 71 attributes are 
related to Johnson Controls administrative data necessary for the management of the bug. 


Product 


Location Skills 


Actors 9 attributes 


Decision | Unclassified 
71 attributes Acknowledge date 


Timing Actual analysis date 
Cost 28 attributes 


2 attributes 


Estimated Cost 


Johnson Controls Detection Means 
Symptom 


Initial criticity bugs' model 2 attributes 
Residual criticity : Severity 


7 attributes Obtained results 
Impact 


Problem 2 attributes 


Solution Type 


14 attributes Origin Phase 
Cause Cause Description 


7 attributes 


Figure 2.18 — Bug’s model currently used in Johnson Controls (this figure is voluntarily 
uncompleted for confidentiality reasons) 


As stated before, some attributes of the bug have free fields in the problems” tracking tool and 
therefore engineers can write anything they want with the main objective of giving as many 
information as possible on the bug. In Figure 2.19, an excerpt of a bug stored in the problems” 
database 1s illustrated. Since the attribute Problem Description has a free field, each engineer 
has the possibility to fill in this field according to her/his reasoning approach. Technical 
language (code variables, electronic and software jargon ...) is often used in such case. In 
fact, there 1s no standard format that engineers must respect when describing a bug. Moreover, 
attributes such as Cause Type and Description are sometimes not filled in. In fact, through 
these attributes, one could identify the responsibility of persons in the problem. 


* In 2001, Johnson Controls buys the electronics business from Sagem Automotive. 
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ATTRIBUTES DESCRIPTION OF THE BUG 


Problem description |from int volume and vehicle speed. (cf. specification) Test 

example. initial Condition : Ignition = 1 CanDatai = 1 

A “very” technical |CanData2 = 1 Vehicle Speed = 0 Wiper_Intermitent = 1 

description ofthe |Obtained Result : Intermitent time = 8s … Then Vehicle Speed = 
problem 20000 (200 km/h)  Obtained Result : Intermitent time = 8s 

Expected Result : Intermitent time = 4s … Tested on E-CAR 


Cause type Not filled in by the engineers 
Cause description Not filled in by the engineers 
Origin phase Development 
Detection Means Validation test 


Figure 2.19 — An excerpt of a bug stored in the problems” database (this figure is 
voluntarily uncompleted for confidentiality reasons) 


Diagnosis 16 


When describing a bug in the problems” tracking tool, there are too many fields to fill in 
(111 attributes), a lot of free fields (about 25%) and a lack of relevant predefined fields (for 
instance, a bug’s typology). As the detection of bugs comes later in the process, engineers 
do not have enough time to fill in all the fields of a bug (missing information). Moreover, in 
case of free fields, an engineer could write anything she/he wants with the main objective of 
giving as many information as possible on the bug (irrelevant information). Since 
information is missing and/or irrelevant, it remains a difficult problem to reuse bugs in 
order to avoid or detect similar problems on future developments. 


C. Existing techniques to reuse capitalized bugs 


We analyze many internal documents related to the reuse of bugs stored in the problems” 
database. We also interview software experts and managers on the current knowledge 
management practices. We come up to the conclusion that bugs stored in the problems’ 
database are rarely used to avoid similar problems on future developments and ensure that 
carmakers will not encounter the same problems on two similar products. 


Actually, no advanced (formal and automated) techniques have been implemented in order to 
reuse stored bugs. Nevertheless, three traditional strategies are currently practiced': 


e Create and update “Lessons Learned Checklist” for software developments. The 
process of creating and updating lessons learnt is illustrated in Figure 2.20. On the one 
hand and once a bug 1s detected on a project, the project leader decides if this bug 
must be verified on other projects or not. The decision process is not formal and is 
mainly based on the experience of the decision maker. In case of a reused bug, this 
bug 1s transferred to the Software Engineering Process Group (SEPG) which confirms 
or not the possible re-use of this bug. The way of describing a bug in the problems” 
database has a major impact on this process. On the other hand, a software forum 
exists where engineers can submit their questions, remarks and recommendations to 
the SEPG which decides or not to generalize the submitted issue. Finally, each reused 
bug or group of bugs and each general issue is summarized in a lesson learnt (a textual 
sentence) to be consulted on future developments. In Figure 2.21, an excerpt of a 
lessons learned checklist 1s 1llustrated. 
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Figure 2.20 — Process of creating and updating “Lessons Learned Checklists” for 


software skill (Mignen 2005) 
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Figure 2.21 - An excerpt of pores learnt to be checked during the design of the 


validation procedure - This figure is voluntarily fuzzyfied for reasons of confidentiality 


(Fradet 2008) 


Use of peer project to check problems on similar products. At the beginning of each 
new project, a list of “similar” previous projects (according to experts) 1s identified. 
Then, engineers have to review all the problems (including software bugs) detected on 
these projects and identify a list of “critical” problems to be checked on the new 
project. 

Increase the reuse of software components from one project to another. À software 
product is composed from a set of software components fulfilling different services. 
Therefore, the reuse of a component from one project to another must be a usual 
process. However, the challenge is to develop components with standard interfaces 
and configurations. The reuse of components is advocated since it increases the quality 
and productivity. Indeed, lessons learnt are already included in the reused software 
components. 


However and according to experts, some bugs occur again from one project to another. In 
order to confirm this citation, we perform a study on the bugs detected on one software 
functionality, the front wiper management functionality, implemented in five different 
projects since 1997 and till 2007. In fact and according to experts, all the projects related to 
the same type of product and car platform of one carmaker have up to 70% of common 
functionalities. In Table 2.11, for each project, we identify the release year of the project, the 
number of Lines Of Code (LOC) implementing the front wiper functionality and the number 
of bugs detected on this functionality. 
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EE 


Project 1 
Project 2 
Project 3 
Project 4 
Project5 


1997 
2001 
2003 
2003 
2007 


the front wiper functionality 
(without comments and blanks) 


3909 
1457 
889 
1255 
1229 


Number of Lines Of Code implementing 


R. AWEDIKIAN 


Number of bugs 


detected on the front 
wiper functionality 


Table 2.11 - Characteristics of the front wiper functionality implemented in five 
different projects since 1997 and till 2007 


Through the analysis of these five projects, we note that the front wiper functionality has, in 
common, 7 features. On some projects, there are one or more additional features that we 
ignored in our study. We make two classifications of the bugs detected on this functionality. 
The first one according to the 7 features (Cf. Figure 2.22) and the second one according to a 
typology of bugs (Cf. Figure 2.23) borrowed from the literature (Beizer 1990, Chillarege 
1992, Grady 1992, IEEE Std. 1044-1993). Then, we make the arithmetic mean by feature and 
by type of bugs of the number of bugs detected on projects 1, 2, 3 and 4. In fact, we try to 
demonstrate that before developing the front wiper functionality on the project 5, we were 
able to predict which feature is the most critical in terms of bugs’ occurrence (feature 3) and 
which types of bugs engineers are vulnerable to (Control flow and sequencing). 
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Figure 2.22 - Classification of the bugs according to the front wiper”’s features 
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Figure 2.23 — Classification of the bugs according to a typology of software problems 
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As a conclusion, the thousands of bugs stored in the problems’ database are not 
systematically nor efficiently reused to avoid or detect recurrent bugs. To better understand 
the reasons for that, we interview software managers who refers to four main issues for 
reusing stored bugs: 


e Manual analysis by experts of the problems’ database is impossible: thousands of 
bugs and 111 attributes by bug. 

Lack of information: 25% of a bug’s attributes are systematically not filled in. 

° _Ambiguous and incomplete information: 25% of the filled attributes are free field and 
these attributes (for instance, problem description) are the most important to detect 
recurrent type of bugs. 

e  Lack of knowledge in data mining processes and tools to perform an automatic 
analysis of the problems’ database. 


Diagnosis 17 


There are no advanced (formal and automated) techniques to reuse bugs stored in the 
problems” database in order to avoid or detect similar bugs on future developments. In fact, 
carmakers are unhappy when encountering the same type of problem on two different 
products delivered by the same supplier. 


VIIL. Managing and reusing test cases in Johnson Controls 


The languages used for designing test cases in each of the unit and validation test activity are 
usually computer languages. Indeed, test cases for unit test are developed in C language and 
test cases for validation test are developed in a script language specific to Johnson Controls. 
A detailed description of these languages 1s provided in Appendix B. Presently, the versions of 
the software components of a project are managed through a commercial version manager 
tool (PVCS*). In consequence, the test cases for unit and validation tests are also versioned 
using this tool and stored in the same folder as the related software component or 
functionality. 


As stated before (Cf. Section 7.C), all the projects related to the same type of product and car 
platform of one carmaker have up to 70% of common functionalities. Therefore, using 
capitalized test cases seems to be beneficial in automotive context. In other words, when 
testing a software functionality that we already implemented in the past on another project, it 
is judicious to reuse existing test cases. Unfortunately, in Johnson Controls capitalized test 
cases are not always reused from one project to another. We interview software experts and 
managers on this phenomenon and we identify two main reasons. The first one is the use of 
different formats to specify a fest case. Sometimes, engineers specify the fest cases 
immediately in the computer language (C language, script language) understandable by the 
test execution platform. Others use the test case format presented in Figure 2.11. In fact, the 
use of computer languages makes the reuse of rest cases a difficult task. One has to analyze 
and adapt fest cases Written in a computer language from one project to another. It is 
important to note that now, festing a software product of about 200 KLOC (Kïilo Lines Of 
Code) requires about 1000 KLOC of tests (Johnson Controls source). The second one is the 
lack of an automated process to reuse test cases. The manual analysis and adaptation of test 
cases from one project to another seems a laborious task. It could be more time consuming to 


0 http:/www.serena.com/products/pvces/pvcs-version-manager.html (Consulted on November 2008). 
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adapt existing test cases than to design new ones. In this situation, the fextual analysis tools 
could help but unfortunately such tools are absolutely not known in the company. 


Diagnosis 18 


Currently, test engineers use different formats to specify a test case. Sometimes, engineers 
specify the test cases in a computer language (C language, script language), others use a 
more high level test case format (independent from the technology). Moreover, there is a 

lack of formal process and tools to manage and reuse test cases from one project to 
another. 


However, one initiative was launched two years ago and had the purpose to create manually 
standard test cases for software validation. An example of a standard test case as it was 
developed at Johnson Controls is illustrated in Table 2.12. 


VPR ID Type of Test Date Modified 
VPR.SPEED.0001.01 Functional 22.03.2006 
Goal Initialization of the pointer 
Applicable if — The device displays the vehicle speed with pointer 
Description of test — The Ignition is switched ON. 


— Set the signal concerning the Speed to value > O0 km/h. 
— The ignition is switched Off 
— Set the signal concerning the Speed to value = 0 km/h. 
— The Ignition is switched ON. 


Expected behavior — At the second ignition ON the pointer should be on its stop 
position. 
Additional Comments — If the project contains 2 or more product lines (ex. Low line, 


High line) repeat the tests on both lines. 


Bug reference (defect ID) 
Table 2.12 —- Example of a standard test case as developed at Johnson Controls 


A set of standard test cases were developed and classified by functionality of product. In fact, 
potential bugs by product functionalities are identified and documented in standard test case 
patterns. These patterns must be systematically consulted (for the given product type) before 
beginning testing stages. This is a conventional RETEX (RETurn of EXperience) strategy, but 
which remains to be completed for any product line. The two main difficulties of such an 
approach are to 1) describe standard test cases With a suitable language level understandable 
by any fest engineer and 2) keep the list of fhese test cases updated without exploding their 
number. Two years after, many issues faced this test case reuse strategy and coerced software 
managers to stop it. The four main issues are 1) the list of standard test cases is no more 
updated due to a lack of resources, 2) an exploding number of standard test cases, 3) all the 
standard test cases are stored in the same Word document which becomes unmanageable and 
finally 4) most of the standard test cases are too much detailed and therefore not 
understandable by a newly-graduated test engineer. 


IX. Conclusion 


Through our industrial audit, We analyze the current software practices at Johnson Controls 
and make diagnoses on the current V&V activities of a software. The performed diagnoses are 
listed in Table 2.13. In Figure 2.24, we locate each of these diagnoses within the Johnson 
Controls software organization. 


In the following chapter, based on our industrial audit, we clearly define the scope of our 
research. We also formulate our research topic in accordance with the research issues in 
software testing. 
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Diagnosis description 
number 


1 Verification and Validation (V&V) practices and testcases are rarely shared betweenthe carmakers and theirelectronic suppliers. 

2 Now, one cannot get a degree in software V&V. Software V&V'is incorporated into the software engineering degree. Moreover, software engineers often prefer development activities compared 
with verification and validation activities. 

3 In automotive industry, semi-formal and formal methods are more and more used to specify software functional requirements. However, there is a lack of a standard formalism shared between 
carmakers and suppliers. In fact, for each project, the supplier has to adaptits processestothe formalism used by the carmaker. 

4 Deadlines for camaker requirements freeze are specified in the cammaker-supplier contract. Nevertheless, the cammaker's requirements evolve continuously along the software development life 
cycle withoutcomplying with these deadlines. Moreover, suppliers mustreact quickly by integrating (without regression) the changes inthe product. 

5 The Software Requirement Specification(SRS) documentis often a large document (about hundreds of pages), difficultto manage, incomplete and notregularly updated. 

6 Sometimes, the SRS document is the official and contractual document between cammaker and supplier. lt is also the main document used by the development and V&V teams in their 
activities. It has a standard structure butthere are no standards to specify carmakers’requirements (more especially functional requirements). 

7 Sometimes, the review of software code is badly done or even ignored. In fact, number of code reviewers is not aware of the coding rules and recommendations to be checked. Moreover, the 


code reviewis not systematically performed on each new software component. In consequence, the code review does not often detectall the bugs that must be detected throughthis activity. 


8 According to the To Be” process, the unit test of a software component must ensure a 100% source code coverage. However, this V&V technique is not responsible to verify the compliance of 
the component's behavior with the carmaker requirements. One software component can be tested unitarily (100% ofcode coverage) without fulfilling the behavior required by the customer. 


9 Sometimes, the unit test of a software component is incomplete or even inexistent. In other words, the source code of the component under test is not covered at 100%. As a consequence, the 
uncovered pieces of code could hide critical bugs. 


When testing a software component or product and after an operation on the input signals, test engineers do not check the behavior of all the output signals of the component or product under 
10 test. Based on their understanding of the program behavior and/or the carmaker requirements, test engineers decide to check only some output signals in relation with the performed operation. 
In fact, they verify the explicitexpected behaviorbutnotthe implicitone. 


11 The present definition of a software requirement is not enough refined. In fact, one requirement can hide two or more implicit requirements. Therefore, inexperienced validators could miss 
testing some of the carmakerimplicit requirements. 

12 In validation test and after selecting an operation to be performed on the software under test, test engineers analyze the carmaker requirements in order to assess the expected values to be 
checkedon some output signals of the software. In fact, this assessmentis based on the engineers’understanding ofthe requirements and may leadto errors. 

13 For each software component or product under test, a large potential operation space is associated. Each engineer has a different perception of the possible and critical operations based on 
his/herexperience. Therefore, the present strategy to select operations in order to test a softwareis irrelevant. 

14 The test cases designed by engineers do not always simulate the real use of the software product under test. The main purpose of testing activities is to cover the software code and 
requirements. As a direct consequence, basic user operations on the product could be nottested by the supplier before a carmaker delivery. 

Presently, the test cases for a software are manually designed by engineers. As the size of automotive software growth, this task becomes a laborious task and accounts for more than 50% of 

the total time and budgetof a project. 


When describing a bug in the problems’ tracking tool, there are too many fields to fill in (111 attributes), a lot of free fields (about 25%) and a lack of relevant predefined fields (for instance, a 

16 bug's typology). As the detection of bugs comes later in the process, engineers do not have enough time to fill in all the fields of a bug (missing information). Moreover, in case of free fields, an 
engineer could write anything she/he wants with the main objective of giving as many information as possible on the bug (irrelevant information). Since infommation is missing and/or irrelevant, it 
remains a difficult problem to reuse bugs in order to avoid or detectsimilar problems on future developments. 


There are no advanced (formmal and automated) techniques to reuse bugs stored in the problems’ database in order to avoid or detect similar bugs on future developments. In fact, carmakers 


Ur are unhappy when encountering the same type of problem on two different products delivered by the same supplier. 


18 Currently, test engineers use different formats to specify a test case. Sometimes, engineers specify the test cases in a computer language (C language, script language), others use a more 
high level test case format (independent from the technology). Moreover, there is a lack of formal process and tools to manage andreuse testcasesfrom one project to another. 


Table 2.13 — List of diagnoses on the software V&V practices in Johnson Controls 


Quality of the design of test cases for automotive software: design platform and testing process 


105 


Industrial audit KR. AWEDIKIAN 


CARMAKER I SUPPLIER |  CARMAKER 
| 


Carmaker | | 


Requirements Cab . Software jelerieine ps ductfrel 
armakbr requirements N requirement software product| -201Ware producilrelease > Acceptance 
specification Î test 


compliance 
Validator 


Requirement engineer 


ab! 5 6 11 


Hardware 


Global design of Integrate 

the software software 
product EE 

Designer Integrator 


SW architecture SW component 


Design software 
components 


Software Verification and Validation techniques 


Code 
review 


Validation 


Static Dynamic 


analysis | analysis | : 14 r 


Integration 


D à 


Test Cases 


= | 
v 
Problems’ Configuration | 
tracking tool management tool | 
E — ë Legend 
L à 
|: Bugs 
de 18 D: Diagnosis i 
TestCases | É number i : 
| n 


database 


Maseassnssedacasandesenssentrnens < 


Figure 2.24 - Localization of the diagnoses within the Johnson Controls software organization 
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CHAPTER 3. RESEARCH TOPIC 
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I. Introduction 


In our research, we go through the Johnson Controls problem with a systemic approach in 
order to identify domains from which we might be able to improve the global performance of 
the software Verification and Validation (V&V) activities. In Chapter 2, we perform an 
industrial audit and make diagnoses on the current V&V practices within the company. Based 
on the industrial audit, one could isolate critical anomalies and lacks in the current engineers” 
practices. À review of related solutions proposed in the literature could help in defining or 
adapting relevant solutions to our context. 


In this chapter, we clearly define the scope of our research and we formulate our research 
topic based on the performed diagnoses and the related research issues in software testing. 
The industrial and academic needs and objectives are summarized in Section 2. A 
specification of our research topic and focus is done in Section 3. Finally, the main software 
testing issues is highlighted in Section 4. In the conclusion of this chapter, we identify the 
diagnoses which are in the scope of our research and associate for each of these diagnoses one 
or more related software testing issues (as stated in the literature). 


IT. Industrial and academic needs and objectives 


A. Initial industrial needs 


Facing the fierce competition within the automotive industry and the strong pressure that 
carmakers impose on their suppliers to reduce the cost and the development time, Johnson 
Controls 1s looking for new and innovative engineering solutions to increase its performances 
and therefore better satisfy the requirements and expectations of their clients. As said in 
Chapter 1 — Section 6, electronic parts represent now up to 30% of the global cost of a car and 
software bugs represent more than 80% of the problems detected on such a product. 
Therefore, decreasing the development cost and increasing the quality of software product 
have become of main interest for carmakers. Johnson Controls as an automotive electronics 
supplier has launched many initiatives (this PhD is one of these initiatives) inside its 
engineering centers around the world with the aims of: 


1. Decreasing the number of bugs detected by the carmaker - Quality 
2. Reducing the development time of electronic projects - Delay 
3. Reducing the development cost of electronic projects - Cost 


Through our research project, we were asked by an automotive electronic supplier namely 
Johnson Controls to improve the performance of its software Verification and Validation 
(V&V) activities. Their main purpose is to improve the quality of their products and therefore 
better satisfy the requirements and expectations of their clients. In Johnson Controls, the 
validation test which is the last V&V activity before a carmaker delivery is considered as the 
ultimate activity to detect all the bugs and therefore deliver carmakers “bug-free” software. It 
represents up to 90% of the time spent in the V&V of a software product (Cf. Chapter 2 — 
Section 5). While the fest cases execution activity is often automated via a fest execution 
platform, the test case design activity remains a manual task that fills in time many engineers. 
Up to 50% of a software project team is dedicated to design fest cases (Cf. Table 1.1). 
Therefore, for many of software managers and experts in Johnson Controls, automating the 
design of test cases seems to be the most adapted solution to reduce the festing time, cost and 
resources while improving the code and requirement coverage. 
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B. Academic objectives 


Since the 1990s, there has been a renewed interest in the development of effective software 
testing techniques. In the years, the topic has attracted increasing interest from researchers, as 
testified by the many specialized events and workshops, as well as by the growing percentage 
of testing papers in software engineering conferences. À recent paper titled “Software Testing 
Research: Achievements, Challenges, Dreams” of a software engineering pioneer named 
Bertolino (Bertolino 2007) organizes the many outstanding research challenges for software 
testing into a consistent roadmap. One of his conclusions was that there is a need to make the 
process of software testing more effective, predictable and effortless. In addition, the author 
pinpoints the many fruitful relations between software testing and other research areas. In 
fact, by focusing on the specific problems of software testing, We may overlook many 
interesting opportunities arising at the border between software testing and other disciplines. 
Unfortunately, few papers exist in which the problem of software testing is considered with a 
systemic approach. 


Our primary scientific goal has been to go through this problem with a systemic approach in 
order to identify levers in any domains from which we might be able to improve the global 
performance of the software V&V activities. The added value of such an approach 1s the 
resolution of the problem with a global quality viewpoint. Therefore, in Chapter 2, we 
perform an industrial audit on the software practices currently used in automotive industry 
and more especially in Johnson Controls. Through the audit, we characterize the overall 
environment of our problem. We understand what verifying and validating a software product 
means and we point out the main current issues and lacks in the automotive V&V activities. 


III. Research scope 


A. Research topic formulation 


Through a primary industrial audit at Johnson Controls, we first analyze the V&V “To Be” 
processes, activities and techniques. Then, we characterize the software engineers’ practices 
(“As Is” processes, activities and techniques) in verifying and validating a software product. 
As a conclusion of the audit, we perform a list of diagnoses on the current V&V practices 
within the automotive industry and more especially automotive electronic suppliers (such as 
Johnson Controls). Based on these diagnoses, our research topic may now be refined through 
these two questions: 


1. How to detect bugs early in the software development life cycle? In other words, 
How to detect bugs closer to where they were introduced? 


2. How to detect “all” the bugs of a software product before a carmaker delivery? 
Or, at least, how to measure that few bugs remain to be found? 


B. Research focus 


Let us define a bit more the exact contour of our research. It was defined in accordance with 
the Johnson Controls priorities and interests. We were not authorized to intervene within the 
software design process in itself to a priori lower the number of bugs. It has been considered 
as another issue. In other words, we do not work on avoiding bugs while designing and 
developing a software product but on detecting the bugs once the product is developed. 


In Chapter 1 — Section 5.C.2, we identify two types of V&V techniques: the sfatic ones and 
the dynamic ones (e.g. software testing). In Chapter 4 — Section 2.B.1, we perform a survey 
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on the sfatic techniques and on how they are adapted or not to the automotive context. Based 
on our industrial audit (Cf. Chapter 2 — Section 5.B), Johnson Controls presently performs 
most of the review static techniques (technical review, walkthrough, inspection and audit). On 
the contrary, the proof static technique is still considered as a non-adapted method to the 
automotive and more especially Johnson Controls context. Even 1f sfatic techniques are 
necessary to detect errors earlier in the development process, they are not sufficient. In fact, 
these techniques focus on analyzing the sfatic product representation and do not test the 
product in its real life (dynamic). This could explain the fact that, in Johnson Controls, V&V 
dynamic techniques are considered as the ultimate techniques to detect all the bugs. They 
represent up to 90% of the time spent in the V&V of a software product (Cf. Chapter 2 — 
Section 5). As a consequence, We focus our research on the V& V dynamic techniques. The 
main dynamic V& V technique is the software testing. 


Testing a software product requires two main activities. À detailed specification of these 
activities 1s done in Chapter 2 — Section 6. The first one consists of designing fest cases and 
the second one of executing these fesf cases on the software product under test. Based on our 
industrial audit within Johnson Controls (Cf. Chapter 2 — Section 5), the execution of test 
cases is performed thanks to Johnson Controls property test execution platforms. These 
platforms are described in Appendix C. On the one hand, the number of bugs related to a 
wrong execution of a fest case is minor regarding the one related to an irrelevant design of a 
test case (1 over 100 — Johnson Controls source). On the other hand, the design of fest cases 1s 
a manual task that accounts for up to 50% of a software project time. Therefore, we focus our 
research on the design of efficient test cases for software. In fact, we are interested in any 
organizational matter that has a positive influence onto the quality of the fest case design 
process: simulation platform, knowledge management, competency management and 
project management. 


IV. Hot research issues in software testing 


The Bertolino’s definition (Bertolino 2003) of the software testing technique highlights the 
four main testing issues (four underlined words in the definition). 


Definition 3.1: Software testing (Bertolino 2003) 


Software testing consists of the dynamic verification of the behavior of a program on a finite 
set of test cases, suitably selected from the usually infinite executions domain against the 
specified expected behavior. 


Dynamic: festing implies executing the program on (valued) inputs. Since sfatic techniques 
(review, inspection .….) are useful to evaluate the internal correctness of a software product, 
testing 1s the only technique allowing the assessment of its behavior when executed in its real 
environment. 


Research issue 1: How to execute test cases on a software product? (This issue is not in 
the scope of our research) 


Finite: even for simple programs, so many test cases are theoretically possible that exhaustive 
testing Would require years to execute. Dijkstra (Dijkstra 1972) calculated that the exhaustive 
testing of a multiplier of two 27-bit integers taking “only” some tens of microseconds for a 
single multiplication would require more than 10000 years. In Chapter 8 — Section 2, we 
demonstrate that testing exhaustively a software product is a NP-Complete problem from a 
computational viewpoint. Generally, the whole test set can be considered infinite. In contrast, 
the number of executions that can realistically be observed must obviously be finite (and 
affordable). Clearly, “enough” festing to get reasonable assurance of acceptable behavior must 
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be performed. This basic need points to well known issues of testing, both technical in nature 
(criteria for deciding to stop testing) and managerial in nature (estimating the effort to put in 
testing). Testing always implies a frade-off between limited resources and schedules, and 
inherently unlimited test requirements. 


Research issue 2: When to decide to stop testing a software product? 


Selected: many operation selection techniques differ on their strategy to select a finite 
number of operations. Test engineers must be constantly aware that different techniques may 
lead to quite different quality results; they also may be much dependent of context factors 
such as the kind of application, the maturity of the process and the organization, the expertise 
of test engineers, the tool platform. How to select the most suitable operations to be 
performed on the software under test is a complex issue (Vegas 2001). 


Research issue 3: How to choose the “relevant” operations to be checked on a software 
product? 


Expected: it must be possible (although not always easy) to decide whether the observed 
outcomes of program execution are acceptable or not, otherwise, the testing would be useless. 
The observed behavior may be checked against specifications and user’s expectations. The 
test pass/fail decision is commonly referred in the testing literature to as the oracle problem. 


Research issue 4: How to assess the expected behavior of a software product? 


V. Conclusion: our diagnoses, the scope of our research and the software 
testing research issues 


Based on our research focus, we identify in Table 3.1 the diagnoses which are in the scope of 
our research (the design of test cases). One diagnosis is related to the static V&V techniques 
and three diagnoses are related to the carmakers’ practices on which a supplier can absolutely 
not act. We also associate for each of the diagnoses in the scope of our research one or more 
related software testing issues (as stated in the literature). 
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Diagnosis In the scope of our research | Related software testing issues 
nn HS AS PHRIeR Ci hu fe 1) Literature review 


Verification and Validation (V&V) NO- related to carmakers' practices 
2 Now, one cannotgeta degree in software V&V... YES Issue 2,3 and4 
a In automotive industry, semi-formaland formal … NO- related to carmakers’'practices = 
4 Deadlines forcarmakerrequirements freezeare .….  NO-—relatedto carmakers’practices © 
5 The Software Requirement Specification(SRS) … YES Issue 2,3 and4 
6 Sometimes, the SRS documentis the official … YES Issue 2,3 and4 
7 Sometimes, the review of software code is NO- related to static V&V techniques = 
8 According tothe To Be” process, the unit test … YES Issue 2 
9 Sometimes, the unit testof a software YES Issue 2 
10 When testing a software component … YES Issue 4 
11 The presentdefinition of a software requirement … YES Issue 2 
12 In validation testand after selecting an … YES Issue 4 
13 Foreach software componentorproductunder … VES Issue 3 
14 The testcases designed by engineers do not … YES Issue 3 
15 Presently, the test cases for a software are MES Issue 3 and4 
16 When describing a bug in the problems'tracking … YES Issue 2 and 3 
17 There are no advanced (formal and automated) … YES Issue 3 
18 Currently, testengineers use different formats … MES Issue 3 


Issue 1 is not in the scope ofourresearch 
Table 3.1 —- Our diagnoses, the scope of our research and the software testing research 
issues 


In the following chapter, we perform a literature review on the existing approaches, 
techniques and tools in the field of the V&V of software products. More especially, we focus 
our research on finding or adapting “solutions” for the anomalies and lacks (diagnoses) that 
we identify via our industrial audit. In fact, we mainly develop the literature related to the 
software testing issues 2, 3 and 4. 


Quality of the design of test cases for automotive software: design platform and testing process 


115 


State-of-the-art KR. AWEDIKIAN 


CHAPTER 4. STATE-OF-THE-ART 


Quality of the design of test cases for automotive software: design platform and testing process 


117 


State-of-the-art KR. AWEDIKIAN 


I. Introduction 


Constructing reliable products continues to be one of software development’s greatest 
challenges. Testing, one of the most crucial tasks along the software development life cycle 
can easily exceed half of a project’s total effort. À successful testing approach can save 
significant effort and increase product quality, thereby increasing customer satisfaction and 
lowering maintenance costs. 


Despite these obvious benefits, the state of software testing practice isn’t as advanced as 
software development techniques overall. In fact, testing practices in industry are, most of the 
time, neither very sophisticated nor effective. This might be due partly to the perceived higher 
satisfaction from developing something new as opposed to testing something that already 
exists. Also, many software engineers consider test engineers as second-class executives. 
They consider testing as a junior or entry position and use it merely as a springboard into 
development jobs. However, academia spends significant effort in researching new testing 
approaches. Promising approaches have started to find acceptance in industry, but the 
technology transfer between festing research and industry 1s still insufficient. Academics 
sometimes say that industry is immature and practitioners are clueless, whereas practitioners 
might argue that researchers squander their time developing cool but useless testing 
technologies. As it often happens, the truth lies somewhere in between. 


In this chapter, we develop the literature related to the software testing issues 2, 3 and 4 and 
we focus our research on finding or adapting “solutions” for the anomalies and lacks 
(diagnoses) that we identify via our industrial audit. An overview on the software verification 
and validation (V&V) techniques is proposed in Section 2. A classification of the software 
testing techniques 1s done in Section 3. Finally, software testing issues and related solutions 
are developed in Section 4. We identify lacks in these solutions and propose improvement 
actions in order to fit in our context. In the conclusion of this chapter, we summarize the 
improvement actions that we propose all along the chapter. 


IL. Verification and Validation of software products 
A. Principles 


Verification and Validation (V&V) of software are defined in the present report after the EEE 
Standard Glossary of Software Engineering Terminology (EEE Std. 610-1990). 


Definition 4.1: Verification and Validation (EEE Std. 610-1990) - Abbreviation: V&V 


The V&V process is the process of determining whether the requirements for a product or 
component are complete and correct, the products of each development phase fulfill the 
requirements or conditions imposed by the previous phase, and the final product or 
component complies with the specified requirements. The distinction between verification and 
validation has been well-framed by Barry Boehm, who memorably described verification as 
“building the product right” and validation as “building the right product”. 


Software V&V helps the product designers and test engineers to confirm that a right product 1s 
build right way throughout the development process and improve the quality of the software 
product. It makes sure that, certain rules are followed when developing a software product 
and also makes sure that the developed product fulfills the required specifications. This 
reduces the risk associated with any software project up to certain level by helping in 
detection and correction of faults, which are unknowingly done during the development 
process. 
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The standard definition of verification is: ‘'Are we building the product RIGHT?" e.c. 
verification is makes sure that the software product is developed the right way. The software 
must confirm to its predefined specifications, as the product development goes through 
different stages, an analysis is performed to ensure that all required specifications are met. 
The verification part of V&V comes before validation and incorporates software inspections, 
reviews, audits, etc. During the verification, the work product (the ready part of the software 
being developed and various documentations) is reviewed / examined by one or more persons 
in order to find and point out the bugs in it. The verification helps in prevention of potential 
bugs. 


The standard definition of validation is: ‘Are we building the RIGHT product?" e.g. a 
software product must do what the customer expects it to do. The software product must 
functionally do what it is supposed to, it must comply with any functional requirement set by 
the customer. Validation occurs at the end of the development process in order to determine 
whether the product complies with specified requirements. Validation starts after verification 
ends (after coding of the product is completed). Testing methods are basically carried out 
during the validation. 


B. Software verification and validation techniques 


Whatever the size of project, software V&V greatly affects software quality. People are not 
infallible, and software that has not been verified has little chance of working. Gibson in 
(Gibson 1992) stated that typically, 20 to 50 errors per 1000 Lines Of Code (LOC) are found 
during development and 1.5 to 4 per 1000 LOC remain even after validation test. Each of 
these errors could lead to an operational failure (bug) or non-compliance with a requirement. 
The objective of software V&V is to reduce software errors to an acceptable level. According 
to Beizer (Beizer 1990), the effort needed can range from 30% to 90% of the total project 
resources, depending upon the criticality and complexity of the software. The V&V techniques 
must be applied at each stage in the software process. It has two major objectives 1) the 
discovery of bugs in a product and 2) the assessment of whether or not the product is useful 
and useable in an operational situation. V&V must establish confidence that the software is fit 
for purpose. This does not mean completely free of defects. Rather, it must be good enough 
for its intended use and the type of use will determine the degree of confidence that is needed. 
Confidence is certainly subjective and depends on many factors such as software criticity, 
users and market expectations. The V&V consists of numerous techniques and tools, often 
used in combination with one another. Due to the large number of V&V approaches in use, we 
cannot address every technique. In fact, software V&V both use static and dynamic techniques 
of product checking to ensure that the resulting software product matches with its 
specifications and that the software product as implemented meets the expectations of the 
customer. In fact, dynamic techniques involve the execution of the software product under 
test, whereas static techniques do not: 


e Static techniques (Review and Proof) are concerned with analysis of the static product 
representation to discover errors throughout all stages of the software life cycle. It may 
be complemented by fool-based document and code analysis. 

e Dynamic techniques (Testing) are concerned with exercising and observing product 
behavior. The product is executed with test data and its operational behavior 1s 
observed. 
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É Static techniques 
a. Review 


A review is a technique during which a work product, or set of work products, is presented to 
project personnel, managers, users, customers, or other stakeholders for comment or approval 
(EEE Std. 610-1990). Review can be used to examine all the products of the software 
evolution process. In particular, they are especially applicable and necessary for those 
products not yet in machine processable form, such as requirements or specifications written 
in lateral language. ZEEE (IEEE Std. 610-1990) has identified four kinds of review which are 
often used for software verification: technical review, walkthrough, inspection and audit. 
These reviews are all “formal reviews” in the sense that all have specific objectives and 
explicit rules of procedures. They expect to identify errors and discrepancies of the software 
regarding the original specifications, plans and standards. 


Technical review 


The objective of a technical review is to evaluate a specific set of review items (e.g. 
document, source code) and provide management with evidence that 1) they conform to 
specifications made in previous phases; 2) they have been produced according to the project 
standards and procedures and finally 3) any changes have been properly implemented, and 
affect only those products identified by the change specification. Typical conclusions of a 
review meeting are 1) authorization to proceed to the next phase, subject to updates and 
actions being completed, 2) authorization to proceed with a restricted part of the product and 
3) a decision to perform additional work. 


Walkthrough 


Walkthrough should be used for the early evaluation of documents, models, designs and code. 
The objective of a walkthrough is to evaluate a specific review item (e.g. document, source 
code). À walkthrough should attempt to identify errors and consider possible solutions. In 
contrast with other forms of review, secondary objectives are to educate, and to solve form 
errors. 


Inspection 


Inspection can be used for the detection of errors in detailed designs before coding and during 
the coding stage. Inspection may also be used to verify test cases. À study done by Fagan 
(Fagan 1986) has shown that inspection could detect over 50% of the total number of errors 
introduced in development stages. EEE (IEEE Std. 610-1990) considers that inspection 1s a 
more rigorous alternative to walkthrough, and is strongly recommended for software with 
stringent reliability, security and safety requirements. 


Audit 


Audit is an independent review that assesses compliance with software requirements, 
specifications, baselines, standards, procedures, instructions, codes and contractual and 
licensing requirements. To ensure their objectivity, audit should be carried out by people 
independent of the development team. 


b. Proof 


A proof attempts to logically demonstrate that software is correct. Whereas a test empirically 
demonstrates that specific inputs result in specific outputs, proof logically demonstrate that all 
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inputs meeting defined pre-conditions will result in defined post-conditions being met. Adrion 
(Adrion 1986) defines proof as 1s a collection of techniques that apply the formality and rigor 
of mathematics to the task of proving consistency between an algorithm solution and a 
rigorous, complete specification of the intent of the solution. This technique 1s also referred to 
as “‘formal verification”. Proof techniques are normally presented in the context of verifying 
an implementation against a specification. 


Prowell and Beizer (Powell 1986, Beizer 1990) have identified several limitations to proof 
techniques. One limitation is the dependence of each proof technique to a formal specification 
language. In fact, in order to use a specific proof technique on a project, the software 
requirements of this project must be written in a specific language associated with the 
corresponding proof technique. Another limitation has to do with the complexity of using 
proof techniques. For large programs, the amount of detail to handle, combined with the lack 
of powerful tools may make the proof technique impractical. According to inner software 
managers, proof techniques are not suitable to the automotive competitive context and more 
especially to Johnson Controls. The four main reasons are: 


e _Proof techniques are often used on critical software products. They often have precise 
and logical specifications with no loopholes and they require being highly reliable, 
since failures in this kind of products may lead to deathly consequences. Some areas 
where proof techniques have been successful are for the specification and verification 
of safe and critical products such as aircraft avionics, nuclear power plant control and 
patient monitoring. In Johnson Controls, the developed electronic products are related 
to the car interior functionalities and are not considered by the carmakers as critical. 

e Automotive engineers are not familiar with proof techniques contrary to aeronautic or 
defense engineers. Software testing is a widespread V&V technique in automotive 
industry. 

e _Proof techniques are not widely used in automotive industry (carmakers and Johnson 
Controls competitors). This could lead to the conclusion that proof techniques are not 
adapted to the automotive context. In fact, the difficulty of expressing software 
requirements in the mathematical form necessary for formal proof has restricted a 
wider application of this technique. 

e  Finally, many managers highlight the complexity and the additional effort required 
regarding reviewing or testing techniques. 


As said in the research focus (Cf. Chapter 3 -— Section 3.B), we do not address the sfatic 
V&V techniques but we focus our research on the dynamic techniques (e. g. software 
testing). 


2. Dynamic techniques 


Software testing, a V&V dynamic technique is a widespread technique in automotive industry. 
In Johnson Controls (Cf. Chapter 2 — Section 5), software testing represents up to 90% of the 
total time spent in verifying and validation a software product. Moreover, in the academic 
research, the traditional focus of software V&V techniques has been the soffware testing. In 
fact, testing approaches are widely studied in academic research and deployed in software 
industry. Therefore, in our literature review, the software testing category has been further 
refined. In the following section, we expose the major festing principles, techniques and 
Issues. 
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III. Software testing techniques 


A. What is “software testing”? 


Harrold (Harrold 2000) has identified several advantages of testing over sfatic-analysis 
techniques. One advantage of testing is the relative ease with which many of the testing 
activities can be performed. Test cases can be generated automatically. Software can be 
instrumented so that it reports information about the executions with the rest cases. This 
information can be used to measure how well the rest cases satisfy the quality objectives. 
Output from the executions can be compared with expected results to identify those fest cases 
on which the software failed. À second advantage of testing is that the software being 
developed can be executed in its expected environment. The results of these executions with 
the test cases provide confidence that the software will behave as intended. À third advantage 
of testing is that much of the process can be automated. With this automation, the fesf cases 
can be reused for testing as the software evolves. Although, testing has a number of 
advantages, it also has a number of limitations. Testing cannot highlight the absence of errors; 
it can only stress their presence. Additionally, testing cannot show that the software has 
certain qualities. Despite these limitations, testing is widely used in industry to provide 
confidence in the quality of software. Therefore, the growth of software complexity and the 
increased emphasis on software quality, highlight the need for improved festing 
methodologies. In the following, we list some citations of software pioneers around the world. 


“Quality assurance over test designs and testing is essential to a successful quality effort. [...] 
More than the act of testing, the act of designing tests is one of the most effective bug 
preventers known. [...] The ideal quality assurance activity would be so successful at this that 
all bugs would be eliminated during test design. Unfortunately, this ideal is unachievable. We 
are human and there will be bugs. To the extent that quality assurance fails to reach its 
primary goal of bug prevention, it must reach its secondary goal of bug detection.” 


B. Beizer, (Beizer 1984) 


“Reliable Object-Oriented software cannot be obtained without testing.” — R.V. Binder 
Binder, (Binder 1995) 


“The importance of software testing and its implications with respect to software quality 
cannot be overemphasized. [...] It is not unusual for a software development organization to 
expend between 30 and 40 percent of total project effort on testing. In the extreme, testing of 
human-rated software (e.g. flight control, nuclear reactor monitoring) can cost three to five 
times as much as all other software engineering activities combined!” 


RS. Pressman, (Pressman 1997) 


However, several different definitions have been given for the soffware testing technique. 
Some of them are listed below. In this dissertation, we adopt the definition proposed by the 
National Institute of Standards and Technology (NIST). 
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Definition 4.2: Software testing (NIST 2002) 


Software testing is the process of applying metrics to determine product quality. Software 
testing is the dynamic execution of software and the comparison of the results of that 
execution against a set of pre-determined criteria. “Execution” is the process of running the 
software on a computer with or without any form of instrumentation or test control software 
being present. “Predetermined criteria” means that the software’s capabilities are known 
prior to its execution. What the software actually does can then be compared against the 
anticipated results to judge whether the software behaved correctly. Software testing is a 
widespread V&V technique in automotive industry. 


Definition 4.3: Software testing (Myers 1979) 


Software testing is the process of executing a program or system with the intent of finding 
errors. 


Definition 4.4: Software testing (IEEE Std. 610-1990, IEEE Std. 829-1998) 
Software testing is: 


(1) the process of operating a software component or product under specified conditions, 
observing or recording the results, and making an evaluation of some aspect of the 
component or product. 


(2) the process of analyzing a software item to detect the differences between existing and 
required conditions (that is, bugs) and to evaluate the features of the software items. 


While the NIST definition associates the software testing to a quality measurement tool, the 
definition of Meyers insists on the fact that testing software must reveal bugs and the ZEEE 
definition claims the good behavior of the software under test. 


B. Classification of software testing techniques 


There is an excess of testing methods and testing techniques. Classified by life-cycle phase, 
software testing can be categorized as follows: unit test, integration test, validation test and 
regression test. Classified by accessibility, software testing can be divided into: white-box test 
and black box test. AI these test methods can be used (individually or in conjunction) at each 
phase of the software life-cycle. In the following, we provide some details on each of these 
testing techniques. 


1. According to life-cycle phase 


During the development lifecycle of a software product, testing is performed at different 
levels and can involve the whole product or parts of it. Depending on the process model 
adopted, then, software testing activities can be articulated in different phases, each one 
addressing specific needs relative to different portions of a product. Whichever the process 
adopted, Bernot (Bernot 1991) states that one can at least distinguish in principle between 
unit, integration and validation test. These are the three festing levels of a traditional phased 
process (such as in Johnson Controls). Pezze (Pezze 1998) considers that these levels are 
complementary with different goals and execution procedures. In fact, none of these levels is 
more relevant or important than the others. Each level must address a specific typology of 
bugs in a software product. The unit test must detect bugs related to the behavior of each 
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software component independently from its environment. The integration test focuses on 
problems of communications and interfaces that may arise during component integration. And 
finally the validation test focuses on the behavior of a software product as a whole. 


a. Unit or component or module test 


A component 1s the smallest testable piece of software (Cf. Definition 1.3), which may consist 
of hundreds sometimes thousands of LOC, and generally represents the result of the work of 
one programmer (developer). Bernot (Bernot 1991) defines the unit test as a V&V technique 
to ensure that a software component satisfies its functional specification and/or that its 
implemented structure matches the intended design structure. The unit test can also be applied 
to check the local data structure (improper typing, incorrect variable name, inconsistent data 
type) and the boundary conditions. In other words, go through all the source code of a 
software component. 


b. Integration test 


Generally speaking, integration is the process by which software components are aggregated 
to create a software product. Bernot (Bernot 1991) defines the integration test as the 
technique that aims at verifying that each component interacts with other components 
according to its specifications. In particular, it mainly focuses on the communication 
interfaces among integrated components. Even though the single components are individually 
acceptable when tested in isolation, in fact, they could still result in incorrect or inconsistent 
behavior when combined. For instance, there could be an improper call or return sequence 
between two or more components. Fenton (Fenton 2000) has identified two integration test 
approach: non-incremental and incremental. In a non-incremental approach the components 
are linked together and tested all at once (big-bang testing). In the incremental approach, we 
find the classical fop-down strategy, in which the modules are integrated one at a time, from 
the main program down to the subordinated ones, or bottom-up, in which the tests are 
constructed starting from the modules at the lowest hierarchical level and then are 
progressively linked together upwards, to construct the whole product. Usually in practice (as 
in Jonson Controls), a mixed approach is applied, as determined by external project factors 
(e.g. availability of modules, release policy, availability of test engineers and so on). 


c. Validation or system test 


Validation test involves the whole software product and 1s defined by Bernot (Bernot 1991) as 
the technique that aims at verifying that the whole software behaves according to the 
customer requirements. In particular it attempts to reveal bugs that cannot be attributed to 
specific components, but they are due to the inconsistencies between components, or to the 
planned interactions of components and other objects (which are the subject of integration 
test). In (Bertolino 2002), Bertolino summarizes the primary goals of validation test: 


e  Discovering the bugs that manifest themselves only at system level and hence were 
not detected during unit or integration test. 

e  Increasing the confidence that the developed product correctly implements the 
required capabilities. 

e  Collecting information useful for deciding the release of the product. 


Validation test must therefore ensure that each product function works as expected. 
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d. Regression test 


In (Bernot 1991), the author consider that the regression test is not a separate level of testing, 
but may refer to the retesting of a component, a combination of components or a whole 
software product after modification, in order to ascertain that the change has not introduced 
new errors. Às software produced today is constantly evolving, driven by market forces and 
technology advances, regression test takes by far the predominant portion of festing effort in 
industry. Since both corrective and evolutive modifications may be performed quite often, to 
re-run after each change all previously executed test cases would be prohibitively expensive. 
Therefore various types of techniques have been developed to reduce regression test costs and 
to make it more effective. Fernandez (Fernandez 1996) proposes a selective regression test 
techniques based on selecting a (minimized) subset of the existing fest cases by examining the 
modifications. Other approaches instead prioritize the test cases according to some specified 
criterion (for instance maximizing the code coverage). 


2: According to accessibility 


Testing methods can also be divided into two families, according to the input data from which 
test cases are selected (Beizer 1990): black-box test and white-box testing. Black-box test, a 
term most likely borrowed from electronic engineers, involves treating software component or 
product as a black-box (like an electronic component) to which input can be supplied and 
from which the corresponding output can be collected and observed, but whose inner 
intermediate workings of the software cannot be seen. White-box test does allow one to 
observe the internal workings of the software and to make use of its structural information to 
adapt or drive the testing process. 


a. Functional or black-box or specification-based test 


According to Beïizer (Beizer 1995), test cases for functional test are derived from the 
functional specification of the software product under test, apart from the code. The criterion 
of correctness 1s the functional specification of the software under test: program behaviors are 
compared to those required by the specification. The goal is to select test cases that cover 
each requirement described by the functional specification. Functional test is typically the 
base-line technique for designing test cases, for a number of reasons. Functional test case 
design can (and should) begin as part of the requirement specification process. Even if the 
source code of a software is not already developed, one can design functional test cases for 
this software based on the software functional requirements. Moreover, functional test is 
effective in finding some classes of bugs that typically elude structural test techniques. 
Functional test techniques can be applied to any description of program behavior, from an 
informal partial description to a formal specification and at any level of granularity, from 
software component to product testing. 


Since, functional test aims at finding any discrepancies between what a software does and 
what it 1s intended to do, one must obviously refer to requirements as expressed by users and 
specified by software engineers. An important side effect of test design 1s highlighting 
weaknesses and incompleteness of software functional requirements. A survey on the 
formalism degree of the software functional requirements 1s performed in Section 4.D.1. 
Designing functional test cases is an analytical process which decomposes requirement 
specifications into fest cases. In most cases (as in Johnson Controls), functional test is a 
human intensive activity. For instance, when test engineers work from informal specifications 
written in natural language, much of the work is in analyzing the specification for identifying 
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test cases. Even expert test engineers can miss important fest cases. Systematic processes 
amplify but do not substitute for skills and experience of the test engineers. In a few cases, 
functional test can be fully automated. This is possible for example when requirement 
specifications are expressed in a formal language (for instance, a grammar or an executable 
model). This approach is known under the name of formal testing or model-based testing and 
has been described by Apfelbaum and Robinson in (Apfelbaum 1997, Robinson 1999). The 
authors highlight the approach’s advantage of guaranteeing a good and formal coverage of the 
requirements specification. In fact, the test engineers” job is limited to the choice of the fest 
selection criteria, which defines the strategy for generating fest cases. Several experiments 
have been performed in festing using formal Specifications. À good summary of these 
experiments has been done by Gaudel and El-Far in (Gaudel 1995, El-Far 2001). 


b. Structural or white-box or program-based test 


According to Beizer (Beizer 1995), test cases for structural test are derived from the code of 
the software under test. In fact, the structure of the software itself is a valuable source of 
information for selecting fest cases and determining whether a set of fest cases has been 
sufficiently thorough. We can check whether a test case has covered a specific part of the 
program. In fact, testing can reveal an error only when the execution of the corresponding 
erroneous items causes a bug. For instance, if there were an error in the line N of the program, 
it could be revealed only with fesf cases that would cause this line to be executed. Based on 
this observation, a program has not been adequately tested 1f some of its items have not been 
executed. An item could be a line of code, decision, condition or procedure (Cf. Chapter 2 — 
Section 6.A.1). Unfortunately, a set of correct program executions in which all structural 
items are exercised does not guarantee the absence of errors. Execution of an erroneous item 
may not always result in a bug. The state may not be corrupted when the item is executed with 
some data values, and a corrupted state may not propagate through execution to eventually 
lead to a bug. Many software researchers (Jorgensen 1995, Pezze 1998, Woodward 2005) 
state that structural information must not be used as the primary answer to the question, “How 
shall I choose tests,” but it is useful in combination with other fest selection criteria such as 
cover the customer requirements. 


Based on our industrial audit (Cf. Diagnosis 8), test engineers in Johnson Controls use 
the structural approach in the unit test stage and the functional approach in the 
validation test stage. The purpose of designing fest cases using the structural approach is 
to cover at 100% the source code while using the functional approach, test engineers 
have to check the compliance of the software with the carmaker requirements. This 
leads to the fact that bugs related to the behavior (regarding the requirements) of one 
independent software component are detected later in the process (during the validation 
test). We propose to perform functional test since the earlier testing stages. One has to 
verify the compliance of each software component (independently from its environment) 
with the carmaker requirements. 


IV. Software testing research issues and solutions 
In this section, we develop the software testing issue identified in Chapter 3 — Section 4. In 


fact, we analyze the related solutions proposed in the literature, identify lacks in these 
solutions and propose improvement actions in order to fit in our context. 
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A. Research issue 1: How to execute test cases on a software product? 


The execution of a test case can occur in a manual or automated Way. In other words, the test 
case descriptions that are the result of the rest design activity could be manually or 
automatically executed against a software product. One issue when automating the fest 
execution 1s to transcribe the specified test cases into a computer language. Another issue 1s 
the ability to put the product into a state from which the specified rest cases can be launched. 
This 1s sometimes referred to as the test precondition. In fact, before a specific command can 
be executed, several runs in sequence are required to put the product in the suitable test 
precondition. An effective way to deal with this is to arrange the selected rest cases into 
suitable sequences, such that each fest case leaves the product into a state similar to the 
precondition of the next fest case. This problem has been early formalized and tackled by 
Dick in (Dick 1993). Moreover, a more complex problem arises when testing only one or 
more components of a software product (the case of a software unit test). Indeed, the testing 
task itself requires a large programming effort. To be able to test one software component of a 
large software product we need to emulate the behavior of its peripheral software 
components. Fortunately, some commercial test tools exist which can facilitate these tasks. 
Finally, when testing reveals a bug, the task of recreating the conditions that made it occur is 
called test replay. Exact test replay requires mechanisms for capturing the happening of 
synchronization operations, and for forcing the same order of operations when a test 1s 
replayed. 


As said in the research focus (Cf. Chapter 3 — Section 3.B), we do not address the 
problem of executing fest cases on the software product. In fact, we made the assumption 
that the present fest execution platforms are reliable. 


B. Research issue 2: When to decide to stop testing a software product? 


Determining when to stop testing and release a product is an important management decision. 
It is clear that there is natural frade-off between the decision to continue testing or to stop: (a) 
if testing stops too early, many bugs remain. Thus we incur the cost of later bug-fixing and 
losses due to customers’ dissatisfaction. The cost of fixing a bug after release 1s a lot more 
than the cost of fixing while testing (Cf. Figure 1.11). (b) if testing continues up to the 
maximum permissible time, then there is the cost of testing effort and a loss of market 
initiative. 


1. Criteria to stop testing a software 
Several stopping criteria have been proposed for software testing. 


a. Stochastic similarity 


À stopping criterion based on stochastic similarity is proposed by Whittaker in (Whittaker 
1994) and refined by Sayre in (Sayre 2000). This criterion is based directly on the statistical 
properties of a usage and testing chains. The usage chain is a model of ideal testing of the 
software; e.g. each arc probability 1s established with the best estimate of actual usage, and no 
failure states are present. The testing chain, on the other hand, is a model of a specific test 
history, including bug data. Thus, the usage chain represents what would occur in the 
statistical test in the absence of bugs, and the testing chain represents what has occurred. 
Dissimilarity between the two models is therefore a useful measure of the progress of testing. 
When the dissimilarity is small, the test history 1s an accurate picture of the usage model. 
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Unfortunately, Johnson Controls test engineers are often subjected to time pressure and 
therefore test and bug data are note often well organized. Therefore, in this context, the use of 
the stochastic similarity model to decide stop testing a software may lead to poor results. 


b. Reliability estimation 


A stopping criterion based on estimated reliability and confidence is proposed by Littlewood 
in (Littlewood 1997). This criterion relies on a target reliability. JEEE (IEEE Std. 610-1990) 
defines software reliability as the probability of “bug-free” software operations for a 
specified period of time in a specified environment. In fact, during the past 30 years, many 
models have been proposed assessing the reliability measurements of software products. A 
software reliability model specifies the general form of the dependence of the bug process on 
the principal factors that affect it: error introduction, error removal, and the operational 
environment. Software reliability modeling forecasts the curve of the bug rate by statistical 
evidences. The purpose of this measure is two-fold: 1) to predict the extra time needed to test 
the software to achieve a specified objective; 2) to predict the expected reliability of the 
software when the festing 1s finished. The success of a model is often judged by how well it 
fits a curve to the observed "number of bugs vs. time" function. It is important to note that all 
the software reliability models are based on some assumptions 1) The module under test 
remains essentially unchanged throughout testing, except for the removal of errors as they are 
found, 2) Removing an error does not affect the chance that a different error will be found, 3) 
"Time" is measured in such a way that testing effort is constant and finally 4) AÏl errors are of 
equal importance. Unfortunately, none of these assumptions fit with our industrial context and 
therefore using such models in deciding when to stop testing a software in Johnson Controls 
may lead to poor results. Many of the current software reliability models, techniques and 
practices are detailed in the Handbook of Software Reliability Engineering by Lyu (Lyu 
1996). 


c. Cost benefit estimation 


A cost-benefit stopping criteria based on estimates of the errors remaining in the product and 
the cost to repair them both before and after release, are proposed by Dalal in (Dalal 1988). A 
more sophisticated version which includes costs due to lost market and customer 
dissatisfaction 1s proposed by Chavez in (Chavez 2000). This model remedies all the 
assumptions considered by the reliability models. However, this leads to a complex 
mathematical problem. Since Johnson Controls test engineers are not familiar with 
mathematical theories (which are not the core of their skills), it remains difficult to apply such 
a model in our context. 


d. Test coverage 


À stopping criteria based on test coverage are presented by Offutt in (Offutt 1999a). The 
decision of when to stop festing is based on covering a software code or requirements in 
various Ways. In practice, code coverage is used to decide when to stop structural test, while 
requirement coverage is used in a functional test context. On the one hand, researches in code 
coverage measurement have reached a high level of maturity and many automated tools were 
commercialized. In a survey done by Yang in (Yang 2006), the author studies and compares 
17 code coverage measurement tools. In fact, the code coverage measurement helps engineers 
detecting “dead code”, piece of code that can be never covered and “non specified code”, 
piece of code that does not implement any of the requirements. On the other hand, 
requirement coverage measurement 1s still immature. In fact, the accuracy of a requirement 
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coverage measurement depends on the degree of formalism used when specifying a set of 
requirements. In Section 4.D.1, we develop the three degrees of formalism (informal, semi- 
formal and formal) used to specify software functional requirements. Measuring the coverage 
of an informal or semi-formal specification 1s usually done by a manual approach (Bontron 
2005). In fact, all the requirements associated to the software product under test are identified 
and when designing a test case, test engineer has to identify the requirement(s) that has (have) 
been covered by this test. Obviously, such an approach is imprecise since it strongly depends 
on the engineers’ degree of the specification comprehension and interpretation. Measuring the 
coverage of a formal specification can be considered as a simple problem that can be easily 
automated (Offutt 1999a). However, the coverage measurement criteria are specifics for each 
formalism of formal specification. Now, in Johnson Controls, the code coverage is formally 
used when festing unitarily each software component. Moreover, in validation test, test 
engineers have to ensure a 100% coverage of the software functional requirements. In 
Chapter 2 — Section 4.A, we show that semi-formal and formal methods are more and more 
used to specify software functional requirements in automotive industry but there is not a 
unique standard formalism shared between carmakers and suppliers. As an indirect 
consequence and even with formal specifications, test engineers still measuring the 
requirement coverage using the tradition manual approach presented in Chapter 2 — Section 
6.B.I. 


Based on our industrial audit (Cf. Diagnosis 2, 5, 6, 8, 9, 11 and 16), the stopping 
criterion used when festing unitarily a software component is the 100% coverage of the 
component source code. Sometimes, for time and budget reasons, test engineers stop 
testing a component even if the 100% code coverage is not reached. In validation test, the 
criterion to stop festing a software product is to cover at 100% the related carmaker 
requirements. These requirements are documented in the SRS document, a large 
document difficult to manage, incomplete and not regularly updated. Moreover, there 
are no standards to specify software requirements and test engineers have to adapt their 
coverage practices to each requirement’s formalism. Finally, the present definition of a 
software requirement is not enough refined. In fact, one requirement can hide two or 
more implicit requirements. Therefore, inexperienced validators could miss testing some 
of the carmaker implicit requirements. Based on the literature review, we consider that 
ensuring a 100% code coverage is a necessary quality objective when festing a software. 
Nevertheless, we propose to formalize the measurement of the requirement coverage. To 
do this, one has to specify the requirements using a formal language. Moreover, we 
suggest integrating project constraints (test time and cost) in the decision to stop testing a 
software product. 


C: Research issue 3: How to choose the operations to be checked on a 
software product? 


Effective testing requires strategies to frade-off between the two opposite needs of amplifying 
testing thoroughness on the one side (for which a large number of fest cases Would be 
desirable) and reducing times and costs on the other (for which the fewer the rest cases the 
better). Given that test resources are limited, how the operations are selected becomes of 
crucial importance. Indeed, the problem of operation selection has been the major dominating 
topic in software testing research. À decision procedure for selecting the operation is 
provided by an operation selection strategy. 


A basic strategy 1s random testing, according to which the operations are randomly chosen 
from the whole input domain according to a specified distribution, e.g. after assigning to the 
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inputs different “weights” (more properly probabilities). For instance the uniform distribution 
does not make any distinction among the inputs, and any input has the same probability of 
being chosen. In contrast with random testing, a broad class of operation selection strategies 
referred to as partition testing. The underlying idea is that the program input domain is 
divided into sub domains within which it is assumed that the program behaves the same, e.g. 
for every point within a sub domain the program either succeeds or fails: we also call this the 
test hypothesis. Therefore, thanks to this assumption only one or few points within each sub 
domain need to be checked, and this is what allows for getting a finite set of operations out of 
the infinite domain. Hence a partition testing strategy essentially provides a way to derive the 
sub domains. An operation selection Strategy vielding the assumption that all operations 
within a sub domain either succeed or fail is only an ideal, and would guarantee that any 
fulfilling set of operations always detect the same bugs; in practice, the assumption is rarely 
satisfied, and different sets of operations fulfilling a same criterion may show varying 
effectiveness depending on how the operations are picked within each sub domain. The 
relative merits of these two different operation selection philosophies have been highly 
debated by Weyuker and Frankl in (Weyuker 1991, Frankl 1998). However, the most 
practiced operation selection strategy in industry is probably based on reaching an objective; 
in particular code and/or requirement coverage objective. In fact, the test engineer keeps 
selecting operations until the predefined objective is reached or her/his manager tells her or 
him to stop (for time or budget reasons). This strategy 1s clearly subjective and based on the 
test engineer's intuition and experience. Nevertheless, expert test engineers can perform a very 
good selection mechanism taking into account many factors such as the time and cost and the 
efficiency of the selected operations (Johnson Controls source). When using this operation 
selection strategy, the test engineer’s skill (experienced and skilled) is the factor that mostly 
affects test effectiveness in finding bugs. 


Many are the factors of relevance when an operation selection strategy has to be chosen. An 
important point to always keep in mind is that what makes a test a “good” one does not have a 
unique answer, but changes depending on the context, on the specific application, and on the 
goal for testing. The most common interpretation for “good” would be “able to detect many 
bugs”; but again precision would require to specify what kind of bugs, as Basanieri has shown 
in (Basanieri 2002) that different operation selection strategy detect different types of faults. 
Paradoxically, operation selection seems to be the least interesting problem for test 
practitioners. In 2006, we did a survey on existing commercial tools supporting the operation 
selection When festing a software product. We focus our survey on the tools able to select 
operations that verify the compliance of a software with its specification (functional 
requirements). We identify 6 tools: 


CONFORMIQ TEST GENERATOR by VERYSOFT - GERMANY 

MATELO by ALLATEC - FRANCE 

PRO-TEST/PRAXIS by DIGITAL COMPUTATIONS, INC - USA 

REACTIS by REACTIVE SYSTEMS, INC — USA 

RHAPSODY TESTCONDUCTOR/AUTOMATIC TEST GENERATOR by 1- 
LOGIX/TELELOGIC - USA 

6. T-VEC RAVE/TESTER for Simulink/Stateflow by T-VEC — USA 


An overview of the characteristics of each of these tools is given in Appendix D. The major 
number of these tools is based on a Model-Based approach. Indeed, the software functional 
requirements are represented in a specific format from which operations are selected 
automatically. On the one hand, more than 278 tools supporting the soffware testing process 
(test management, test execution and so on) have been referenced in (Legeard 2007) by 
Legeard. On the other hand, we have shown through our industrial audit (Cf. Chapter 2 — 
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Section 6) that the activity of designing manually test cases for software products becomes 
more and more laborious and time consuming. Therefore, one could highlight the lack of tools 
supporting the selection of operations when festing a software. 2% (6 over 278) of the 
commercial software testing tools are dedicated to the software testing activity that accounts 
more than 50% of the total software project time and budget. 


Based on our industrial audit (Cf. Diagnosis 2, 5, 6, 13, 14, 15, 16, 17 and 18), the 
operation selection strategy currently used in Johnson Controls is a manual subjective 
one based the test engineers’ experience and intuition. Their main purpose is to reach a 
code or requirement coverage objective. In fact, test engineers do not always select 
operations that simulate the real use of the software product under test. Moreover, there 
is no formal process to analyze recurrent bugs stored in the problems’ database and 
select operations that detect these bugs on future developments. And finally, there is a 
lack of formal process and tools to manage and reuse fest cases from one project to 
another. We propose to formalize the process of selecting operations in order to be 
independent as much as possible from the test engineers’ experience. One solution we 
propose is to automate this process. One could select operations randomly or, select 
operations based on the end-user behavior”’s profile or the experience feedback (from bugs 
and fest cases capitalized on similar projects in the past). 


Since 2005 and as the design of test cases for software has reached the 50% of the total time 
and budget of a project in Johnson Controls, the automation of the fest case design process 
became a hot topic. Therefore, inner software experts and managers have evaluated some of 
the previous listed tools (evaluation version of the tool). None of these tools are fully adapted 
to the Johnson Controls context. Many issues have been identified by the experts and 
managers: 1) these tools propose to represent the software requirements in a formal language 
which is not adapted to the automotive software context, 2) these tools do not propose a 
relevant stop testing criterion based on the fest case quality and software project constraints 
(test cost and time), 3) some of these tools do not manage the reuse of capitalized bugs and 
test cases from one project to another ,4) some of these tools do not propose to generate test 
cases With a end-user behavior's profile and finally 5) the tool licenses and trainings are 
expensive. 


1. Advantages and drawbacks of automating the design of test cases 


The activity of designing test cases for a software product is a major activity in a software 
development life cycle. To cut down cost of manual test case design and to increase reliability 
of it, researchers and practitioners have tried to automate it. Many managers today expect test 
design automation to be a silver bullet; killing the problems of test scheduling, the costs of 
testing, defect reporting, and more. However, there are many factors to consider when 
planning for test design automation. It usually has broad impacts on the organization such as 
the skills needed to design and implement automated tests, automation tools, and automation 
environments. Development and maintenance of automated tests is quite different from 
manual tests. The job skills change, test approaches change, and testing itself changes when 
automation 1s installed. These impacts have positive and negative components that must be 
considered. Automation 1s only a means to help accomplish our task — testing a product. It 
may reduce staff involvement during testing, thus saving time relatively to manually 
designing test cases. But, automatic test design may generate a bunch of results that can take 
much more staff involvement for analysis, thus costing more than manual test design. Often 
the information obtained from automatic test generation is more cryptic and takes longer to 
analyze and isolate when bugs are discovered. In fact, successful test automation efforts don’t 
focus on eliminating the test team, they focus on doing a more effective and efficient job of 
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testing with the human resources available. Automatic test generation can be incredibly 
effective, giving more coverage. It also provides us with opportunities for festing in ways 
impractical or impossible for manual testing. Indeed, automatic test design can generate 
millions of test cases limited only by the machine power and time available for running the 
tests. However, Black (Black 2000) notice that automated testing 1s a huge investment, one of 
the biggest that organizations make in testing. Tool licenses are often expensive. Engineers 
cannot use alone most of these tools and therefore training, consulting, and expert contractors 
can cost more than the tools themselves. Moreover, the test engineers could resist using an 
automation tool since they felt that their manual process worked fine. Effort must be invested 
in incorporating a new automation tool into the process. As we propose to automate the 
design of fest cases Within Johnson Controls, we must take into account all these 
considerations. 


2. Design test cases based on end-user behavior”’s profile 


We propose to define an end-user (driver) behavior’s profile for each software under test. 
Therefore, when testing the software, one could select the operations or succession of 
operations recurrently performed on the software in real use. In fact, there is no better way to 
test a product other than testing it in the way that it will be used. The main work in this field 1s 
the one of Musa in (Musa 1993). Musa has proposed a process to define an operational 
profile for a system. This process involves one or more of the following five levels: Client 
type list, User type list, System modes, Functional profile and Operational profile. By 
customizing this process to our context, we only consider the operational profile level. In fact, 
we consider that a software product is dedicated to only one customer (carmaker) and a client 
type list is not necessary. The end-users (driver) of an automotive software product can be 
classified regarding to criteria such job, climate, sexe, age, culture .. In our research, we do 
not deal with these criteria and we consider a nominal end-user behavior’s profile. Most 
software products have more than one mode of operation (normal mode, factory mode and 
diagnostic mode). However, the occurrence probability of the normal mode is about 99% 
(Johnson Controls source) and consequently we decide to ignore the other modes. The next 
step 1s to break system modes down into the functionalities. It needs, to create a functionality 
list in determining the occurrence probability of each functionality. The best source of data to 
determine occurrence probabilities is usage measurements, e.g. frequency measurements of 
the users operations, taken on the last release or on similar system. In our context, we don’t 
have this type of information (since it 1s considered as confidential by the carmakers). Finally, 
for each functionality, a set of operations and sucession of operations 1s possible (Cf. Chapter 
2 — Section 6). Therefore, for each functionality, experts must identify recurrent operations 
and succession of operations and therefore define occurrence probabilities. Barnaghan 
(Branaghan 1999) has developed the fundamentals of wsability testing. In fact, the usability 
testing techniques are widely and often used in testing Graphical User Interfaces (GUT). 


3. Design test cases based on experience feedback 


We also propose to reuse capitalized bugs and test cases from one project to another. 
Therefore, when testing the software, one could use the experience feedback on bugs and test 
cases respectively detected and designed on similar software in the past. Presently, 
information on stored bugs is missing and/or irrelevant and reusing these bugs in order to 
avoid or detect similar problems on future developments remains a difficult problem (Cf. 
Diagnosis 16). Therefore, classifying stored bugs and identifying the recurrent type of bugs 
detected on a specific type of software could be useful. In the next section, we perform a 
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survey on the bug classification models. We propose to define a typology of bugs adapted to 
our context and useful to focus the design of est cases on recurrent type of bugs. 


a. A survey on the software bugs classification models 


In this section, we present one bug classification scheme proposed by ZEÉE standard and two 
industrial schemes: the Hewlett-Packard Scheme (HP) and the Orthogonal Defect 
Classification Scheme (ODC) developed by ZBM. Classification is performed by assigning a 
set of measurement variables (attributes) to discrete values, which are selected, based from a 
predefined set of values (attribute values). Therefore, bug classification schemes can differ in 
the way different attributes or attributes values relate to each other. 


Orthogonal Defect Classification (Chillarege 1992) 


The ODC scheme has been developed by IBM. Since its definition, this classification has been 
adopted by more and more organizations. In a survey performed in 1999 by Paulk in (Paulk 
2000), 14 out of 37 high-maturity software organizations (according to the CMM maturity 
model) used this scheme as quantitative analysis practice. The aftributes of this scheme are 
organized according to two process steps: 


e Step OPEN: when a bug has been detected and a bug report is opened in the bug 
tracking system. 
e _ Step CLOSE: when the bug has been corrected and the bug report is closed. 


Two interesting attributes was taken into account in this scheme: 1) the attribute defect type 
which captures the fix that was made to resolve the bug and 2) the attribute trigger which 
captures the reason why an error turns into a bug. The entire ODC scheme with attribute 
name, meaning and values is described by Chillarege in (Chillarege 1992). 


Hewlett-Packard Scheme (Grady 1992) 


The HP scheme was developed by HP’s Software Metrics Council in 1986. This scheme 1s 
based on three descriptors for each bug: 


e The origin — where was the bug introduced in the product 

e The type of the bug 

e The mode — whether information was missing, unclear, wrong, changed or done in a 
better way 


The choice of an attribute value for the attribute Origin defines the possible set of attributes 
available for the attribute Type. The entire HP scheme with attributes and attribute values 1s 
developed by Grady in (Grady 1992). 


IEEE Standard Classification for Software Anomalies (IEEE Std. 1044-1993) 


The /EEÉE scheme was developed by the /nstitute of Electrical and Electronics Engineers 
(EEE), the world's leading professional association for the advancement of technology. The 
different attributes of the scheme are organized according to a general bug classification 
process consisting of four steps: 


First Step: Recognition — the bug is found 

Second Step: Investigation — We identify issues and propose solutions 

Third Step: Action — we establish a plan of action to resolve the problem 

Last Step: Disposition — we complete all required resolution actions and long-term 
corrective actions 
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The entire EEE scheme with attributes and attribute values is developed in (EEE Std. 1044- 
1993). 


b. Major aspects of a software bug model 


The previous survey reveals several valuable elements to be take into account when designing 
a new bug classification scheme. Bugs are inserted due to a particular reason into a 
particular piece of software at a particular point in time. The bugs are detected at a specific 
time and occasion by noting some sort of symptom and they are corrected in specific way. 
Each of these aspects might be relevant for a specific measurement and analysis purpose. 
Mellor and Fenton in (Mellor 1992, Fenton 1996) have proposed a framework of bug key 
elements that capture on high-level aspects of a bug. Each of these key elements can be 
refined leading to many attributes that can be captured by means of measurement: 


e Location: where in the product? The location of a bug describes where in the product 
the bug was detected. This attribute can also contain attribute values describing 
different high-level entities of the entire product (Specification, Design, Code, 
Documentation ..….). 

e  Timing: when in the process phases, we introduce, detect and correct the bug? The 
timing of a bug refers to process phase when the bug was created (origin phase), 
detected (detection phase) and corrected. 

©  Symptom: what we observe when the bug occurred? Symptom captures what was 
observed when the bug occurred or the activity revealing the bug. For instance, the 
ODC attribute Trigger captures the mechanism that allows a bug to occur. Under 
symptom it is also possible to classify what is observed during diagnosis or inspection. 
For instance, in /EEE classification scheme, the aftribute symptom provides a 
classification of the symptom. 

e End result: what are the impacts of the bug on the company itself, on the customer, on 
the end-user? End result describes the failure caused by the bug. For instance, in 
ODC, the attribute impact captures the impact of a bug on the customer (performance, 
usability, instability ..….). 

e Mechanism: in which activity and how, we introduce, detect and correct the bug? 
Mechanism describes how the bug was created, detected and corrected. Creation 
describes activity that inserted bug into the system. Detection describes activity that 
was performed when the bug was detected (code review, unit test ...). Correction 
refers to the steps taken to remove the bug. 

e Cause: What is the mistake that leads to the bug? Cause describes the mistake leading 
to the bug. For instance, in (Mays 1990) the author uses attributes values like 
Education, Oversight, Communication, Tools and Transcription for an attribute Cause. 
In (Leszak 2000), the author uses different afttributes capturing different kind of 
causes: Human-related Causes (lack of knowledge, communication problems ..….), 
Project Causes (time pressure, management mistake) and Inspection Causes (no or 
incomplete inspection, inadequate participation .….). 

e  Severity: what is the severity of the bug? Severity describes the severity of a resulting 
or potential failure on the whole behavior of the product. 

e Cost: How much the bug cost the company? Cost captures the time or effort to locate, 
isolate and correct an error. 


Bug classification scheme often have problems including incomplete, ambiguous and 
overlapping attributes and attribute values. To prevent such problems, a bug classification 
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scheme needs to be well defined. Freimut in (Freimut 2001) has proposed a list of quality 
properties of a good bug classification scheme: 


e  Orthogonal attributes and orthogonal attributes values: This means that for a 
particular bug and for each attribute only one attribute value is appropriate. If the 
attribute values are not orthogonal, it may happen that two or more attribute values 
may fit so that the engineer has arbitrarily to decide which value to assign. This leads 
to inconsistent and unreliable data. 

e Complete attribute values: The set of attributes value must be complete so that for all 
bugs an appropriate attribute value can be selected. If the set of values are not 
complete, engineer may decide not to classify the bug or select the nearest possible 
value. 

e Small number of attribute values: The scheme must contain a small number of 
attributes values, as too large a number can make selection of the appropriate attribute 
value difficult and therefore unreliable. 

e Clear meaning and definition of attributes and attribute values: The attributes and 
attribute values of the scheme need a clear definition. This definition has to be 
developed with all engineers who have to use the aftributes and need an understanding 
of the attribute. 


D. Research issue 4: How to assess the expected behavior of a software 
product”? 


An important component of testing is the oracle. Indeed, a test is meaningful only if it is 
possible to decide about its outcome (“OK” or “Not OK”). The difficulties inherent to this 
task, often oversimplified, had been early articulated by Weyuker in (Weyuker 1982). In 
much of the research literature on software testing, the availability of oracles is either 
explicitly or tacitly assumed, but applicable oracles are not described. The research literature 
on test oracles is a relatively small part of the research literature on software testing. Some 
older proposals (Panzl 1978, Chapman 1982) base their analysis either on the availability of 
pre-computed input/output pairs or on a previous version of the same program, which 1s 
presumed to be correct. The former hypothesis is usually too simplistic: being able to derive a 
significant set of input/output pairs would imply the capability of analyzing the product 
outcome. In the current industrial practice of software testing, the oracle is often a human 
being. Relying on a human to assess program behaviors has two evident drawbacks: accuracy 
and cost. While the human “eyeball oracle” has an advantage over more technical means in 
interpreting incomplete, natural-language specifications, humans are prone to error when 
assessing complex behaviors or detailed, precise specifications, and the accuracy of the 
eyeball oracle drops precipitously with increases in the number of test runs to be evaluated. 
Even if it were more dependable, the eyeball oracle is prohibitively expensive for large 
volumes of fest cases, and so may become a limiting factor when other parts of testing are 
accelerated with automation. Therefore, automated oracles could be a well adapted solution 
to this problem. Baresi’s (Baresi 2001) survey proposes approaches to automate the test 
oracles. In view of these considerations, it must be evident that the oracle might not always 
judge correctly. So the notion of relevance of an oracle is introduced to measure its accuracy. 
Bertolino in (Bertolino 1997) proposes to measure the oracle accuracy by the probability that 
the oracle rejects a test, given that it must reject it. 


Based on our industrial audit (Cf. Diagnosis 2, 5, 6, 10, 12 and 15), the oracle currently 
used in Johnson Controls is a human being. In fact, after selecting an operation to be 
performed on a software, test engineers analyze the source code and/or the carmaker 
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requirements of this software in order to assess the expected values to be checked on 
some output signals. In fact, this assessment is based on the engineers’ understanding of 
the code and/or requirements and may lead to errors. Moreover, as automotive software 
becomes more and more complex, this task becomes a laborious task and accounts for 
more than 50% of the total fime and budget of a project. We propose to automate the 
assessment process of all the expected outputs values by developing a simulation model 
of the software functional requirements. In fact, test engineers could perform the 
selected operation on the requirements model and assess the output values automatically 
by simulating the model. Moreover, once developing a simulation model of the software 
requirements, one could formally measure the requirement coverage. 


1. Modeling and simulation of software functional requirements 


a. Types of software requirements 


In software domain, several standards organizations (including the Z£EE) have identified four 
categories of requirements: 


e  Functional requirements are the main customer requirements. They refer to the 
behavior of the product. For instance, in a body controller product”! , the behavior of 
the front wiper management functionality is specified by a set of functional 
requirements. 

e Non functional requirements are the interface requirements between functionalities 
and software performances in terms of CPU load and memory capacity. An example of 
non functional requirements can be the communication protocols. 

e  GUI(Graphical User Interface) requirements are the customer requirements related to 
user interfaces. This category of requirements 1s frequent in electronic display product. 

e Non technical requirements include all organizational customer requirements. 
Confidentiality, return of experience, past defects reviews capitalization is examples of 
these requirements. 


Johnson Controls has adopted this typology of requirements (Cf. Definition 2.6). As 
demonstrated in Chapter 2 - Section 4, the functional requirements account for more than 
90% of the carmaker requirements related to the software domain. Therefore, through our 
research project, we focus on the software functional requirements and how one could verify 
the compliance of a software product with its functional requirements. 


b. Formalisms in specifying the functional requirements of a software product 


Both Dart and Brinkkemper in (Dart 1987, Brinkkemper 1990) propose same definitions of 
informal, semi-formal and formal specification: 


e _Informal: These techniques do not have complete sets of rules to constrain the models 
that can be created. Natural language (written text) and unstructured pictures are 
typical instances. 

e Semi-formal: These techniques have a defined syntax. Typical instances are 
diagrammatic techniques With precise rules that specify conditions under which 


*1 A body controller module is an automotive electronic module in charge of managing all electrical currents of a 
car 
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constructs are allowed and textual and graphical descriptions with limited checking 


facilities. 


Formal: These techniques have rigorously defined syntax and semantics. There is an 
underlying theoretical model against which a description expressed in a mathematical 
notation can be verified. Simulation languages are typical instances. 


In Table 4.1, Duphy (Duphy 2000) propose a list of advantages and drawbacks for each of the 
informal, semi-formal and formal formalisn. 


RE RE 7 VE 


Informal 


Semi-formal 


Formal 


Easy to be understand by all the project actors. 
No training but need to understandthe writing 
rules. 


Graphical and abstract representation. 

Easy to be understood by allthe project actors. 
Synthetic, structuring and intuitive 
representation. 

Modularity and reuse. 

Better traceability. 


Precision. 

Abstraction levels. 

Formalism. 

Model verification and validation. 

Proof. 

Simulationis possible. 

Generation of code and test cases from the 
specification model. 

Avoid imprecision, ambiguities and 
contradictionsinthe customer requirements. 
Long term cost reduction. 


Ambiguity. 

Incompleteness. 
Inconsistency. 

Imprecision. 

Cannotbe easily automated. 


Lackin precision. 

Sometimes, notationsare ambiguous. 

Difficultto be interpreted. 

Simulation not possible. 

No techniques to verify and validatethe model. 
Code generation from these models are not reliable. 


Trainings are mandatory. 

Ifno trainings, cost and delay increase. 

Not easy to be understood. 

Used for critical software products. 

Lack in tools supporting formal methods. 

Rarely used in industry. 

Not integrated to the software development process. 


Table 4.1 - Advantages and drawbacks of informal, semi-formal and formal specification 
languages (Duphy 2000) 


In Table 4.2, Duphy (Duphy 2000) has evaluated these three formalisms based on four 
criteria: modelling precision, use, communication facility and training cost. 


1994). 


Informal 
Semi-formal + ++ 


Formal +++ s 


Modelling Communication | Training cost 
precision facility 
Fi +++ +++ ++ 


++ + 


+++ very positive; ++ positive; + quite positive; - quite negative; --negative;--- very negative 
Table 4.2 - Evaluation of the informal, semi-formal and formal specification languages 
(Duphy 2000) 


In Table 4.3, a classification of the specification languages is proposed by Fraser in (Fraser 
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Natural Language Specifications UML Finite State Machines 
(Although oftenused, published Variations on Data/ControlFlowDiagrams  Statecharts 
studies are less numerous) Entity-Relationship Diagrams Markov Chain 
DeMarco Decision Tables 
Gane and Sarson PetriNets 
PSL/PSA Executable Specifications 
SADT GIST 
SERM Refine 
IORL VDM 
CORE Anna 
SDL Z 
JSD CSP 
GIST 


Predicate-Transition-Nets 


Table 4.3 - Classification of the specification languages (Fraser 1994) 


In fact, we cannot talk in detail about all the specification languages. In (Davis 19838), the 
author discussed a variety of informal, semi-formal and formal languages useful for testing. In 
the Section 4.D, we propose to develop a simulation model of the software functional 
requirements in order to automate the test oracle and formalize the measurement of the 
requirement coverage. Only formal languages could be used to simulate software functional 
requirements (Cf. Table 4.1). Therefore, we focus our research on the most useful (according 
to the literature) formal languages: Finite State Machines (FSM), Statecharts, Markov Chains 
and Decision Tables (DT). An FSM is a hypothetical machine that can be in only one of a 
given number of states at any specific time. In response to an input, the machine generates an 
output and changes state. Both the output and the new state are purely functions of the current 
state and the input. FSMs are applicable to any model that can be accurately described with a 
finite number (usually quite small) of specific states. Chow in (Chow 1978) was one of the 
earliest researchers addressing the use of FSMs to specify the behavior of a software. Now, 
there is work on FSMs in software engineering with varied tones, purposes, and audiences 
(Apfelbaum 1997, Robinson 1999, Liu 2000). Sratecharts, extensions to FSMs, were 
proposed by Harel in (Harel 1987). Statecharts make it even easier to model complex real- 
time system behavior with less ambiguity. The extensions provide a notation and set of 
conventions that facilitate the hierarchical decomposition of FSMs and a mechanism for 
communication between concurrent FSMs. Statecharts are probably easier to read than FSM5, 
but they are also nontrivial to work with and require some training upfront. A sample of 
software requirements expressed using Statecharts has been proposed by Hong in (Hong 
2000). Markov chains are stochastic models proposed by Kemeny in (Kemeny 1976). They 
are structurally similar to FSM and can be thought of as probabilistic automata. In fact, a 
probability is associated for each transition. The sum of the probabilities associated to the 
transitions that get out of the same state must be equal to 1. Many researchers worked on 
using the Markov chains in specifying the behavior of a software (Whittaker 1994, Walton 
2000). Sometimes there 1s a need to describe the required external behavior of some aspect of 
a system when the FSM approach makes no sense. One simple solution is the Decision Table 
proposed by Moret in (Moret 1982). À DT is used to lay out in tabular form all possible 
situations on the inputs of a system and to specify which action to take on the outputs in each 
of these situations. 
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c. How to choose a language to specify the functional requirements of a software 
product? 


Many researchers (Davis 1988, Sommerville 1997, El-Far 2001) state that there are no 
software specification languages today that fit all intents and purposes. In fact, for each 
context decisions need to be made as to what language (or collection of languages) is most 
suitable. No large-scale studies have been made to verify the claims of any particular 
language. However, in his paper (Davis 1988), Davis has identified five criteria to help 
choosing the most adapted specification language for a specific context: 1) understandable to 
non computer-oriented customers, 2) used as the main input for the development and 
validation teams, 3) automated checks for ambiguity, incompleteness, and inconsistency, 4) 
encourage the requirements engineer to think and write in terms of external product 
behaviour, not internal product components, and finally 5) provide a basis for automated 
source code and test generation. 


Based on the study that we performed on the evolution of the formalisms used by carmakers 
to specify functional requirements related to software (Cf. Chapter 2 — Section 4.A), we 
underline the increase of formal languages and the decrease of informal and semi-formal 
languages. However, within the formal languages, there not a unique standard formalism 
shared between carmakers and suppliers (70% of FSMs and Sfratecharts and 20% of DTs). In 
fact, for each project, the supplier has to adapt its processes to the formalism used by the 
carmaker. Duphy (Duphy 2000) highlights many works that try to complete or combine semi- 
Jormal and formal languages in order to have the fully, consistent and reliable view of a 
software. Three categories of combination can be identified: 1) create a new formalism based 
on the existing techniques concepts, 2) complete an existing technique with the aim of 
reducing its weakness and finally 3) use simultaneously several existing techniques and thus 
cumulate their advantages. In our context, we propose to develop a new formal (simulation) 
specification language better adapted to the Johnson Controls context. In fact, for each 
project, we propose to represent the software functional requirements of the carmaker into this 
language. In order to be able to represent all the carmaker requirements (now and in the 
future), 1t could be judicious to base our specification language on a combination between the 
FSM (Statechart) and the DT languages; the two languages that carmakers tend to use. 


V. Conclusion 


We believe that the importance placed on testing will increase as software’s pervasiveness in 
everyday life increases. Our dependence on software, from driving cars to shopping on the 
Internet, will decrease users’ tolerance of defective software. Although testing isn’t the only 
software engineering practice to ensure quality software, it remains an essential component of 
the software development’s life cycle. We focus our research on the design of efficient test 
cases for improving the quality of software products. In fact, we are interested in any 
organizational matter that has a positive influence onto the quality of the test case design 
process: simulation platform, knowledge management, competency management and project 
management. 


In this chapter, we pinpointed the main progress in each of these fields when designing test 
cases. Many techniques and approaches have been developed and for each one, we identify 
the advantages and drawbacks to be used or adjusted to our context. As a conclusion, we 
propose a list of actions that could improve significantly the global performance of the 
Johnson Controls company: 
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e  Perform functional test since the earlier testing stages. One has to verify the 
compliance of each software component (independently from its environment) with 
the carmaker requirements. 

e  Formalize the measurement of the requirement coverage. To do this, one has to 
specify the requirements using a formal language. Moreover, we suggest integrating 
project constraints (test time and cost) in the decision to stop testing a software 
product. 

e  Formalize the process of selecting operations in order to be independent as much as 
possible from the test engineers’ experience. One solution 1s to automate this process. 
One could select operations randomly or, select operations based on the end-user 
behavior’s profile or the experience feedback (from bugs and test cases capitalized on 
similar projects in the past). 

e Automate the assessment process of all the expected outputs values by developing a 
simulation model of the software functional requirements. In fact, test engineers could 
perform the selected operation on the requirements model and assess the output values 
automatically by simulating the model. Moreover, once developing a simulation model 
of the software requirements, one could formally measure the requirement coverage. 


Based on our proposals, in the following four chapters (Chapter 5, 6, 7 and 8), we start 
specifying our approach to improve the global performance of the Johnson Controls V&V 
activities. Firstly, we develop a new simulation model of the software functional 
requirements. Secondly, we provide methods and tools to verify and validate the requirements 
model. Thirdly, we propose to monitor the generation of test cases by quality objectives and 
cost constraints. And finally, we suggest refining the operation space description With the 
driver behavior’s profile, past bugs and test cases. 
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CHAPTER 5. MODELING AND SIMULATION 
OF SOFTWARE FUNCTIONAL REQUIREMENTS 
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I. Introduction 


Ten years ago, formal methods were rarely used in automotive industry, contrarily to medical, 
avionics and railways industries. The main argument of automotive industry managers was 
the high cost of deploying and using formal methods. But, as automotive electronic products 
becomes more and more complex, automotive industry 1s required to start adapting existing 
formal methods to their context or developing new ones. Actually, the cost of non-quality 
(warranty and customer dissatisfaction) exceeds the cost of using formal methods. We still 
have to change the engineers’ practices and even adapt the education in software engineering 
to the challenge of complex products. Now, in automotive industry, semi-formal and formal 
methods are more and more used to specify software functional requirements (Cf. Diagnosis 
3). However, there is a lack of a standard formalism shared between carmakers and suppliers. 
In fact, for each project, the supplier has to adapt its processes (test case design, requirement 
coverage measurement) to the formalism used by the carmaker (Cf. Chapter 2 — Section 6.B). 
Most of the automotive electronic suppliers use the SRS (Software Requirement Specification) 
model (Cf. Chapter 2 — Section 4.E). This model is mainly used to organize by functionality 
and by type the carmakers’ requirements related to software and to tag them. 


In this chapter, we develop our new formal language to model software functional 
requirements (a simulation model). Advantages and drawbacks of using formal languages in 
modeling software functional requirements are summarized in Section 2. Our formal model to 
represent software functional requirements is developed in Section 3. We identify two types 
of software functional requirements in automotive industry. These types of requirements 
could be modeled using a Decision Table element or a Finite State Machine element. The 
simulation process of the requirements model is described in Section 4. Finally, a didactic 
case study is proposed in Section 5 in order to better illustrate our requirements model. 


In the following, we use the shortcut “software specification” to designate the “software 
functional requirements specification”’. 


IX. Advantages and drawbacks of formal languages in modeling software 
functional requirements 


Formal specification 1s a specification expressed in a language whose vocabulary, syntax, and 
semantics are formally defined, and which has a mathematical, usually formal logic and basis. 
In this dissertation, we adopt the definition of a formal specification language proposed by 
Wing in (Wing 1990). 


Definition 5.1: Formal Specification Language (Wing 1990) 


A formal specification language provides a formal method’s mathematical basis. ..…. A formal 
specification language provides a notation (its syntactic domain), a universe of objects (its 
semantic domain), and a precise rule defining which objects satisfy each specification. 


Carmakers consider different standards to express the software functional requirements of a 
given electronic module. Based on the study performed in Chapter 2 — Section 4.A, some 
carmakers still use semi-formal and informal methods, but most of them start using formal 
methods (simulation models). Incompleteness and ambiguity are the main characteristics of 
informal and semi-formal methods (Cf. Chapter 4 — Section 4.D.4). More than 30% of the 
bugs detected on a software product are related to lacks in and incomprehension of software 
functional requirements (Johnson Controls source). In fact, when designing test cases for the 
validation test, test engineers should assess the expected values to be checked on the output 
signals of the software product under test (Cf. Chapter 4 — Section 4.D). Indeed, a test is 
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meaningful only 1f it is possible to decide about its outcome. In case of an informal or a semi- 
formal representation of functional requirements, the assessment of the expected values on 
output signals is always a human being (Cf. Chapter 2 — Section 6D, as in Johnson Controls). 
Relying on a human to assess program behaviors has two evident drawbacks: accuracy and 
cost. In fact, after choosing the future operation to be performed on the software product, test 
engineers analyze the customer requirements and identify the required behavior to be checked 
on the product. In case of large volumes of test cases or complex behaviors, the accuracy of 
the eyeball oracle drops precipitously. However, in case of a formal representation of 
functional requirements, the assessment of the expected values on output signals could be 
done automatically (by simulation). The use of formal specification methods is expected to 
lead to increased software quality and reliability. 


Hall (Hall 1990) suggests that benefits of using formal specifications are obtainable without 
an increase in, and possibly in lowering, development costs. However, Sommerville 
(Sommerville 1997) indicates that formal specification methods have not been widely 
accepted in industrial software development. Nevertheless, a number of strategies have been 
proposed for incorporating formal specification methods into the software development 
process. 


On the one hand, a variety of advantages has been attributed to the use of formal software 
specifications. These advantages include understanding of specifications, help in the 
verification of specifications and automatic generation of the source code and test cases. 
Firstly, according to Wing (Wing 1990), the formal specifications help crystallize the 
customer’s vague ideas, and reveal or avoid contradictions, ambiguities, and incompleteness 
in the specifications. Sommerville (Sommerville 1997) highlights that, depending on the 
formal specification language used, it may be possible to animate (simulate) a formal system 
specification to provide a prototype system. The simulation model can be used by inner 
engineers and by end-users to gain further insights into the behavior of the specified system. 
Secondly, as formal specifications can be analyzed using mathematical operators, many 
researchers (Wing 1990, Kemmerer 1990, Fraser 1994) propose to use mathematical proof 
procedures to test (and prove) internal consistency and correctness of specifications. 
Furthermore, the completeness of the specifications can be checked in the sense that all 
enumerated options and combinations have been specified. Thirdly, from an implementation 
point of view, as the final problem solution -the implementation- will be in a formal language 
(e.g. programming language); it is easier to avoid misconceptions and ambiguities in crossing 
the divide from formal specifications to formal implementations. This raises the possibility of 
automatic code generation from formal specifications and therefore avoiding the manual and 
labor coding of the software. Moreover, formal specifications can be used as a guide to the 
test engineers of software components in identifying and generating automatically appropriate 
test cases. In our research project, we do not consider the code generation aspect. /n 
conclusion, the use of formal methods can lead to higher-quality specifications, 
implementations and testing. 


On the other hand, a number of reasons by various authors have been suggested to explain the 
lack of using formal methods in industrial contexts. Firstly, Leveson (Leveson 1990) 
pinpoints the lack of methodological and support tool in formal specification research which 
makes it difficult to develop, analyze, and process large-scale specifications using formal 
specification languages. Secondly, Sommerville (Sommerville 1997) highlights that the 
notation and the conceptual grammar of formal specification languages require familiarity 
with discrete mathematics and symbolic logic which most practicing software engineers do 
not currently have. Thirdly, the very formality which makes formal specifications desirable 
during the later phases of software specification makes them an inappropriate tool for 
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communicating With the end-user during the earlier requirement elicitation and confirmation 
stages. Finally, Sommerville (Sommerville 1997) suggests that management is generally 
conservative and unwilling to use new techniques whose benefits are not yet established. 
Given these difficulties in using formal methods, challenges remain in integrating formal 
methods with the system development effort and in scaling up formal method techniques to 
large-scale real-world development projects. 


In Table 5.1, we summarize the main advantages and drawbacks of formal languages in 
modeling/specifying software functional requirements. 


Advantages 


Avoid contradictions, ambiguities, and Lackof methodological and supporttool 
incompleteness in the software 
specifications 


Automatically Check consistency, Lack of familiarity with discrete 
correctness and completeness of software mathematics and symboblic logic that most 


specifications practicing software engineers do not 
currently have 

Automatically generate code for software Inappropriate tool for communicating with 

product the end user during the earlier 
requirements elicitation and confirmation 
stages 

Automatically generate test cases for Need to verify and validate the developed 

software functional testing model. Indeed, we have to proove the 
conformity between carmaker 


requirements and the developed model 
Reduce development costandtime 


Easy maintenance 


Table 5.1 - Advantages and drawbacks of formal languages in modeling/specifying 
software functional requirements 


Unfortunately, within the formal languages currently used in automotive industry, there 1s not 
a unique formalism shared between carmakers and suppliers (Cf. Diagnosis 3). In Chapter 4 — 
Section 4.D.4, We pinpoint the benefits of a unified formal (simulation) language able to 
model all types of software functional requirements. 


HI. Our formal language to model software functional requirements for 
functional simulation 


Nowadays, and according to Davis and El-Far (Davis 1988, El-Far 2001), an international 
unified model to specify and simulate software functional requirements doesn’t exist. After 
studying a variety of models in literature, we came up with the fact that each model has been 
developed for a specific industrial or academic context. Based on the study that we performed 
on the evolution of the formalisms used by carmakers to specify functional requirements 
related to software (Cf. Chapter 2 — Section 4.A), we underline that, within the formal 
languages, there is not a unique standard formalism shared between carmakers and suppliers 
(70% of FSMs and Sfatecharts and 20% of DTs). 


In our research project, we define our own formal model, to represent software functional 
requirement, keeping in mind the automotive context and its constraints. As defined before 
(Cf. Definition 2.4, 2.5 and 2.6), a software functionality is described by some features that 
are described by some requirements. In the following, we do not consider the non-functional 
requirements and we focus our research on modeling software functional requirements. 
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A. Typology of software functional requirements 


Each software functionality has a set of configuration (Config), input (1), output (O) and 
intermediate (Int) signals with discrete domains. These signals interconnect the features (F) 
of the functionality and each feature is composed from one or more requirements of the same 
type. Based on our study of the carmakers’ requirements related to software (Cf. Chapter 2 — 
Section 4.A) and the literature review on modeling software specifications (Davis 1988, 
Apfelbaum 1997, Robinson 1999), we identify two types of software functional requirements: 


e  combinatorial (Cf. Figure 5.1) if the values of the requirement output signals at 
instant f (O_Reg;) depend on the sole values of the requirement input signals at instant 
t (L Reg). 


l_Req, | O Req 


Req 


© Regq,= f(l_Regq;) 
Figure 5.1 — “Combinatorial” functional requirement 


e  Sequential (Cf. Figure 5.2) if the values of the requirement output signals at instant 
(O_Regq,) not only depend on the values of the requirement input signals at instant t 
(L_ Reg) but also on the values of the requirement output signals at instant 1-1 (O_Reg, 


D). 


l_Req Req | 
© Regq,=f(l_ Reg, O_Reg,.;) 
Figure 5.2 — “Sequential” functional requirement 


_O Reg 


In Figure 5.3, we provide a graphical illustration of our unified functional requirements 
model. This example is the functional requirements model of a software functionality which 
has 1 configuration signal, 3 input signals, 4 output signals, 5 intermediate signals and 4 
features. Configuration signals allow to parameterize the software functionality (for instance, 
by activating or deactivating one feature). Input signals could be switches, sensors or car 
environment variables (for instance, the vehicle speed signal). Output signals could be 
actuators or any type of command (for instance, the wiper motor command signal). Finally, 
intermediate signals allow to manage and share data between two or more features. 


Configuration signals Intermediate signals 


Feature 
N°1 


Feature 
N°4 


& À FeatureT 
sl. N°2 


{clockB 


Input signals Output signals 
Figure 5.3 - Graphical illustration of our unified formal model to represent software 
functional requirements 


Quality of the design of test cases for automotive software: design platform and testing process 


150 


Modeling and simulation of software functional requirements R. AWEDIKIAN 


A “Clock” signal (Cf. Figure 5.4) is required since the behavior of a software product is ruled 
by synchronism. In fact, a “Clock” is just a signal that alternates between zero and one, back 
and forth, at a specific pace (cycle time). It sets the “pace” for the functional simulation of the 
model. The value of this “cycle time” depends on some timing characteristics of the software 
functional requirements. It should be defined by the modeler once analyzing and designing 
the requirements model. 


o| u Time 
Cycle time 
Figure 5.4 - The shape of a “Clock” signal 
B. Two types of modeling elements to model the features of a software 
functionality 


As we stated before, each feature is composed from one or more functional requirements of 
the same type (combinatorial or sequential). We propose to model these two types of 
functional requirements thanks to two types of modeling elements. 


1. Decision table element (DT) 


Moret and Chvalovsky (Moret 1982, Chvalovsky 1983) were the first to thoroughly explore 
the uses and capabilities of DT. We use a DT element to model a feature composed from one 
or more combinatorial functional requirements. A DT is a table (Cf. Figure 5.7) that presents 
a set of exclusive conditions on the DT input signals (Cq) and their corresponding set of 
actions on the DT output signals (Ag). Each set of conditions (Cq) represents a requirement in 
a DT element. The characteristics of a “condition” and an “action” on a signal (S;) are 
respectively illustrated in Figure 5.5 and Figure 5.6. 


The structure of a “Condition” is organizedas following: 
Condition(Operator Op, String Sig, float Val) 


{ 
Operator = Op; / ANY, EQUAL, NEQUAL, GREATER, LESS, GREATER EQ, LESS EQ 
Signal = Sig; 
Value = Val; 


} 


Rules of defining a “Condition” on asignalS;: 
1- When Operatoris setto ANY, then Signal mustbe setto “’”’and Value mustbe setto O 
2- When Operatoris differentfrom ANY and Signalis different from “” then Value must be setto O0 


Examples: 
S,: Condition(ANY, “”, 0) << no matter the value ofthe signals, 


( 
S;: Condition(LESS, "S;”, 0) < if the value ofthe signalS, is LESS than the value of the signalsS, 
S3: Condition (EQUAL, “”, 10) <> if the value of the signal, is equalto 10 
S,: Condition(GREATER_EQ,“”, 5) «if the value ofthe signalS, is GREATER than or EQUAL to 5 


Figure 5.5 — Characteristics of a “Condition” 
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The structure of a “Action” is organized as following: 
Action (GlobalOperator GOp, float Vali, float Val2, Operator Op, String Sig1, String Sig2) 
{ 
GlobalOperator = GOp; / UNCHANGE, EQUAL 
Signali = Sigi; 
Signal2 = Sig2; 
Operator = Op; / NONE, ADD, SOUS, DIV, MULT 
Value1 = Vali; 
Value2 = Val2; 


} 


Rules of defining an “Action” on a signalS;: 
1- When GlobalOperatoris set to UNCHANGE, then Signal1 and Signal2 mustbe setto “” 
Operator mustbe setto NONE and Value1 and Value2 mustbe setto © 


2- When GlobalOperatoris set to EQUAL and Operatoris setto NONE, then Signal2 mustbe set 
to “’and Value2 mustbe setto 0. Andif Signall is different from “” then Value1 mustbesetto 0 


3- When GlobalOperatoris set to EQUAL, Operatoris differentfrom NONE and Signal and 


Signal? are different from “”, then Value1 and Value2 mustbe setto 0 


4- When GlobalOperatoris set to EQUAL and Operatoris different from NONE, Signal is 
differentfrom “and Signal2 is equalto “ ”, then Value2 mustbe setto 0 


5- When Operatoris equalto DIV and Signal2is differentfrom “”, thenthe value of the Signal2 
mustbe differentfrom 0 


6- When Operatoris equalto DIV and Signal2 is equalto “”, then Value2 mustbe different from 0 


Examples: 

S;,: Action (UNCHANGE, “”, “”, NONE,0, 0) no actions to do ons, 

S;,: Action (EQUAL, “S.”, “”, NONE, 0, 0) <> S, mustbe setto the value ofthe signals, 
S;: Action(EQUAL, “”, “”, NONE,, 5, 0) = S, must be set to 5 


S:Action(EQUAL, ‘S,”, “”, SOUS, 5, 0) > S, mustbe setto the value of(S, -5) 
S: Action (EQUAL, “”, “”, MULT, 5, 10) > S, mustbe setto the value of (5 x 10) 
S,:Action(EQUAL, ‘S,”,"S,”, DIV, 0,0) > S; must be set to the value of (S; / S;), with S; # 0 


( 
( 
( 
S,:Action(EQUAL, ‘S,”, “S;”, ADD, 0, 0)  $, must be setto the value of (S, + S:) 
( 
( 
( 
S: Action (EQUAL, *S;”, “”, DIV, 8, 0) = S, mustbe set to the value of (S, / 3) 


Figure 5.6 — Characteristics of an “Action” 


As said before, each software functionality has a set of configuration (Config), input (1), 
output (O) and intermediate (Int) signals. These signals interconnect the features (F) of the 
functionality. In fact, an input signal of a Decision Table element could be a configuration, 
input or intermediate signal of the functionality. While an output signal of a Decision Table 
element could be an output or intermediate signal of the functionality. À Decision Table 
element is illustrated in Figure 5.7, a. For one set of conditions (for example, C1 in Figure 
5.7), it must require that at least one input of the DT is set to a specific value (/7=1), the other 
inputs of the DT may be indifferent (ANY). 
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DT DT 
INPUT SIGNALS 


EC EUR non 00 


Requirements 
Conditions 


n0, nf, n2, n3, n4, n5, q and n ARE integers *= (ANY, “”, 0) 
** = (UNCHANGE, “”, “”, NONE, 0, 0) 


Figure 5.7 — A Decision Table element 


Let us consider the DT element illustrated in Figure 5.8. This DT has 2 input signals and 2 
output signals: 11, Domain = {0, 1}; 22, Domain = {1, 2, 3}; O1, Domain = {0, 1}; O2, 
Domain = {0, 1}. When designing this DT element, designers did not consider all the possible 
conditions on the input signals of a DT (3 out of 6 possible conditions, Cf. Figure 5.8a). They 
only identify the conditions (Ci) which were explicitly specified in the customer requirements. 
In fact, when dealing with a small DT, all the possible conditions can be easily identified. In 
Figure 5.8b, we illustrate the exhaustive DT of the one of Figure 5.8a. On the one hand, a 
condition could be splitted into 2 or more conditions With the same actions on the output 
signals (C1 to C1.1, C1.2 and C1.3). On the other hand, some conditions do not have any 
impact on the output signals (C4, UNCHANGE). However, in industrial context, the problem 
is à little bit more difficult since the number of the DT input signals can exceed 10 and the 
domain length of one signal can exceed 100 (for instance, when sampling the “vehicle speed” 
signal). In that case, its remains a very difficult task to identify manually all the possible 
conditions and their corresponding actions. Therefore, an automatic generation of all the 
possible conditions on the input signals of a DT could be judicious. One could develop a 
computer macro able to generate automatically an exhaustive list of conditions for a DT. 
Unfortunately, we do not have enough time to develop this macro and in our experiments (Cf. 
Chapter 10), we design manually exhaustive DTs. 


(a): Notexhaustive (b): Exhaustive 


DT DT 
INPUT SIGNALS en Ana INPUT SIGNALS 


Actions 


2] o 
= € 
.Q © 
= = 
T Le) 
= € 
© [°} 
O O 


ER 


UNCHANGE UNCHANGE 


Figure 5.8 — Not exhaustive vs. exhaustive Decision Table element 
2. Finite State Machine element (FSM) 


Gill (Gill 1962) introduces FSM theory in 60’s. Since, many applications (Chow 1978) such 
as in software engineering have been performed. We use a FSM element to model a feature 
composed form one or more sequential functional requirements. In our case and in addition to 
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the input and output signals of a FSM, each FSM can have a timing signal (FSMTempo) and a 
set of internal signals (FSMInt,,). The timing signal helps to model timing requirements and 
the internal signals characterize the states of a FSM. A graphical illustration of a Finite State 
Machine element is illustrated in Figure 5.9. It is composed from: 


e an initial state (S0) and a finite number of states (Si) with a set of actions (Ai) on the 
FSM output, internal and timing signals. The FSM timing signal is set to O0 each time 
the sfate of the FSM changes. In fact, the FSM timing signal computes the time spent 
in each sfate. 

e a set of transitions (Tij) from original sfate (Si) to destination sfate (Sj), and for each 
transition (Tij), a set of exclusive conditions (Cij,g) on the FSM input, internal and 
üming signals. Each set of conditions (Cij,q) represents a requirement in a FSM 
element. 


For one set of conditions (for example, Cij,1 in Figure 5.9), it must require that at least one 
input signal of the FSM or one FSM internal signal or the FSM timing signal is set to a 
specific value (7/7=1), the other signals of the FSM may be indifferent (ANY). 


FSM FSM 
INPUT SIGNALS INTERNAL SIGNALS 


Requirements 
Conditions 
ConfignO 
FSMintn3 
+] FSMTempo 


FSM 
INTERNAL SIGNALS TIMING SIGNAL 


‘” NONE, 10, 0) (EQUAL, “”, “”, NONE, O, 0) 


FSMint,, …, FSMint,,; are FSM internal signals 
FSMTempo is the FSM timing signal 
Figure 5.9 — A graphical illustration of a Finite State Machine element 


Let us consider the FSM element illustrated in Figure 5.10. This FSM has 2 input signals, 2 
output signals: 11, Domain = {0, 1}; 22, Domain = {1, 2, 3}; O1, Domain = {0, 1}; O2, 
Domain = {0, 1}. When designing this FSM element, designers did not consider all the 
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transitions and conditions. They only consider the transitions (Tij) and conditions (Ci) which 
were explicitly specified in the customer requirements (6 out of 8 fransitions and 7 out of 9 
conditions, Cf. Figure 5.10a). In order to be exhaustive when designing a FSM, modelers 
must identify all the transitions and conditions that get out of a state even if they do not allow 
to change the sfate of the FSM (Cf. Figure 5.10b). 


(a): Notexhaustive (b): Exhaustive 


States | Actions | OUTPUT SIGNALS 


T12 C1 =0 ANY 


T20 ci ANY | =3 = 
ci ANY | =1 = 

T23 = 
c2 ANY | = 

T31 ct =1 | ANY 


Figure 5.10 — Not exhaustive vs. exhaustive Finite State Machine element 


The functional simulation process of our software requirements 
model 


A synchronized functional simulation can be performed on our model of software functional 
requirements. The simulation is done with an oriented acyclic logic going from input to output 
signals of the software functionality. To better illustrate the simulation mechanism, let us 
consider the example of the Figure 5.3. The simulation order of the features has to be defined 
when designing the model (Feature 1 then Feature 2 then Feature 3 then Feature 4). The 
“Clock” signal synchronizes the behavior of the functional model. Indeed, at each cycle time, 
all the features are simulated following the predefined order. Simulating a feature consists of 
assessing its output signals values according to its input signals values. 
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In case of a feature modeled using a Decision Table element, all conditions (Cq) have to be 
checked. There is no specific checking order for these conditions since up to one condition 
can be fulfilled at a time. Values on the DT output signals are updated according to the action 
associated to the satisfied condition. Note that, in some cases, none of the conditions (Cg) can 
be fulfilled and therefore no actions (Ag) have to be done on the DT output signals. In fact, 
the DT conditions do not often consider all the possible combinations between the values of 
all the DT input signals. Let us consider the DT element of the Figure 5.8a. In Figure 5.11, a 
simulation scenario of this DT is shown. 


e (Figure 5.11a): After initialization, II is set to O and /2 is set to 2. On the next front 
edge of the “Clock” signal, all the conditions (Ci) are checked following the 
predefined order. Once a set of conditions is satisfied (C1), the corresponding actions 
(A1) on the DT output signals are performed and the conditions checking is stopped. 

e (Figure 5.11b): 11 is set to 1. On the next front edge of the “Clock”, C3 is satisfied. 

e (Figure S5.11c): 12 is set to 3. On the next front edge of the “Clock”, none of the 
conditions is fulfilled. 
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Checking order 


Figure 5.11 —- An example to illustrate the simulation process of a Decision Table 
element 


In case of a feature modeled using a Finite State Machine element, one state must always be 
activated. When simulating a FSM, all conditions of all the transitions that get out of the 
activated state have to be checked. There is no specific checking order for transitions and 
conditions since they are exclusive and up to one condition (transition) can be satisfied 
(crossed) at a time. Therefore, after each FSM simulation, at maximum one transition is 
crossed. The origin sfate of the transition 1s deactivated, the destination sfate 1s activated and 
values on output signals are updated. However, in some cases, none of the fransitions that get 
out of the activated state can be satisfied and therefore the activated state remains the same 
and no actions have to be done on the FSM output signals. In fact, the conditions of all the 
transitions that get out of the same sfate do not often consider all the possible combinations 
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between the values of all the FSM input, internal and timing signals. Let us consider the FSM 
element of the Figure 5.10a. 1n Figure 5.12, a simulation scenario of this FSM is shown. 


(Figure S.12a): After initialization, II is set to 1 and 72 is set to 7. On the next front edge of 
the “Clock” signal, all the conditions (Ci) on all the fransitions (T0)j) that get out of the 
activated sfate (S0) are checked following the predefined order. Once a set of conditions 1s 
satisfied (701, C1), the corresponding transitions (T01) is crossed, the origin state (SO) is 
deactivated, the destination sfate (S1) is activated and the action (A1) on the destination state 
1s performed. 


(Figure 5.12b): the activated state is S1. 12 is set to 2. On the next front edge of the “Clock”, 
all the conditions (Ci) on all the fransitions (T1j) that get out of the activated sfate (S1) are 
checked following the predefined order. Since, none of these conditions is satisfied, the 
activated state does not change (S7) and the values on the FSM output signals no more. 


(Figure 5.12c): the activated sfate is SI. II is set to 0. On the next front edge of the “Clock”, 
the transition (T12) is crossed. The new activated state is (S2). 


(Figure 5.124): the activated state is S2. 12 is set to 1. On the next front edge of the “Clock”, 
the transition (T23) is crossed. The new activated state is (S3). 
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Figure 5.12 —- An example to illustrate the simulation process of a Finite State Machine element 
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V. À case study on modeling software functional requirements using our 
new formal and simulation language 


In this section, we develop a simple case study in order to illustrate how an engineer can 
design a simulation model of a set of software functional requirements. To do this, we 
consider a functionality (“Auto_Light”) which has 3 configuration signals, 5 input signals and 
2 output signals: 


e  Configl (“Auto_Light_Config”), Domain = {0, 1} 

e  Config2 (“Follow_Me_home_Config”), Domain = {0, 1} 

e _ Config3 (“Follow_Me_home_Calib”), Domain = {5, 10, 15} 
e 11 (“Reset”), Domain = {0, 1} 

° 22 (“Luminosity_Sensor”), Domain = {0, 1, 2, 3, 4,5, 6,7} 
e 15 (“Car_Locked”), Domain = {0, 1} 

e 14 (“Ignition”), Domain = {0, 1} 

e 15 (“Light Combi_Switch”), Domain = {0, 1} 

e O1 (“Head_Lamp”), Domain = {0, 1} 

° O2(“Tail_Lamp”), Domain = {0, 1} 


The software functional requirements of this functionality were specified by the carmaker 
using the natural language (Cf. Figure 5.13). In fact, this functionality can be decomposed 
into 3 features: Feature 1, Feature 2 and Feature 3. 
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Feature 1 (“Luminosity_Level_Calculation”) 


Req1.1: In case of “Luminosity_ Sensor” is equal to 1 or 2, then “Luminosity_Level” 
mustbe equal to 1 


BReq1.2: In case of “Luminosity_ Sensor” is equal to 3, 4 or 5, then “Luminosity_Level” 
mustbe equal to 2 


Req1.3: In case of “Luminosity_ Sensor” is equal to 6 or 7, then “Luminosity_Level” 
must be equal to 3 


Reqg1.4:in other cases, “Luminosity_Level” mustbe equal to O 


Feature 2 (“Follow Me Home Mode”) 


Req2.1:_In case of “Car Locked” is equal to 1 and “lgnition” is equal to O0, then 
“Follow _ Me Home Activate” mustbe equal to 1 


Req2.2:in other cases, “Follow _Me_Home_Activate” must be equal to 0 


Feature 3 (“Head _Tail_ Activation”) 


Reg3. 1:Once “Reset” is set to 1, “Head_Lamp”and“Tail_Lamp” mustbe set at 0 


Req3.2: In case of “lgnition” is equal to 1 and “Light Combi_Switch” is equal to 1 and 
“Auto_Light_ Config” is equal to 1 and “Luminosity Level” is equal 1 or 2, then 
“Head_Lamp” mustbe equal to 0 and“Tail_Lamp” must be equal to 1 


Req3.3: In case of “lgnition” is equal to 1 and “Light Combi_Switch” is equal to 1 and 
“Auto_Light_Config” is equal to 1 and “Luminosity_ Level” is equal 3, then “Head Lamp” 
must be equal to 1 and “Tail_Lamp” mustbe equal to O 


Req3.4:_ if ‘“Head_Lamp” is equal to 1 or “Tail_Lamp” is equal to 1 and in case of 
“Follow_Me_Home Activate” is equal to 1 and “Follow _Me_ home Config” is equal to 1, 
then “Head_Lamp” mustbe equal to 0 and“Tail_Lamp” mustbe equal to 1. 


f“gnition” is set to 0, then Regq3.5 and Req3.6 


Req3.5: In case of “Head_Lamp” was equal to 1, wait “Follow Me home _Calib” ms 
andthen set“Head_Lamp” and “Tail_Lamp” to 0 


Req3.6: In case of “Head_Lamp” was equal to O, than wait “Follow_Me_home_Calib”/2 
ms andthen set “Head_Lamp”and“Tail_Lamp” to O 


Figure 5.13 — The software functional requirements of the functionality ‘“‘’Auto_Light” as 
they were specified by the carmaker 


Once analyzing the requirements of Figure V13, we came up to the conclusion that Feature 1 
and Feature 2 can be modeled using Decision Table elements and Feature 3 can be modeled 
using à Finite State Machine element. We also identified two intermediate signals: Int1 
(“Luminosity_Level”) and /nf2 (“Follow_Me_Home_Activate”). A graphical illustration of 
the requirements model of the functionality “Auto_Light” is developed in Figure 5.14. 
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« Auto_Light » 


Configi=Auto Light Config 


Config2=-Follow_ Me _ home _Config Ecdturos 
Config3=Follow_Me_home_Calib FSM1 
I=Reset Ot-Head Lamp 
12=Luminosity_Sensor Feature 1 dnosity_Level 
DT O2=Tail Lamp 
13=Car_Locked | 
Feature 2 dw_Me Home _Activa 
l4=Ignition DT 2 


15=Light_Combi_Switch 


Figure 5.14 — A graphical illustration of the requirements model of the functionality 
‘“Auto_Light” 


The Decision Table elements of the Features 1 and 2 are developed in Figure 5.15 and Figure 
5.16. These DT are exhaustive. 


Feature 1- DT 1 


Our model Br DT? 
Carmaker req ID req ID Conditions | __INPUTS Actions | OUTPUTS 
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4 carmaker requirements splitted into 8 requirements in our model 


Figure 5.15 - Feature 1 modeled using a Decision Table element 


Feature 2 - DT 2 
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2 carmaker requirements splitted into 4 requirements in our model 


Figure 5.16 — Feature 2 modeled using a Decision Table element 


A graphical illustration of the Finite State Machine element of the Feature 3 is developed in 
Figure 5.17. This FSM is not exhaustive. Figure 5.18 details the sfates, transitions and 
conditions of this FSM. In fact, the Feature 3 has a sequential behavior since the behavior of 
the signals O1 and O2 depends not only on the signals Config1, Config2, Config3, 11, 15, Int1 
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and /nf2 but also on the current state of the feature. We also define a FSM timing signal 
(FSMITempo) and one FSM internal signal (FSMIInt1, Domain = {0, 1}) in order to model 
the requirements Reg3.5 and Reg3.6. 


Feature 3- FSM 1 


A1 
Config1= O1=0 
Auto_Light Config O2=0 
FSM1Int1=0 
Config2= 
Follow _Me_ 
home_Config 
C1:Configi= 
Config3= 
Follow _Me_ 
home _Calib ot= 
Head _Lamp 
I1=Reset 
T30 
C1:F$M1inti=1, O2= 
l4=ignition, | FSM1Tempo> Config3/2 = ce À e : Tail Lamp 
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15= FSM1Tempo> Config3 
Light Combi_Switch T12 
C1:Config1=1,l4=1,15=1, Inti=3 
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Luminosity_Level A4 
Int2= 
Follow Me _ 
Home _Activate 
1, Inti=1 
C1:Config1=1,14=1,15=1, Inti=3 1, Inti=2 
Figure 5.17 - Feature 3 modeled using a Finite State Machine element — Graphical 
illustration 
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6 carmaker requirements splitted into 17 requirements in our model 


Figure 5.18 — Feature 3 modeled using a Finite State Machine element — States, 


Transitions and Conditions 
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Even with instructions and guidelines, we are conscious that two different modelers can 
design two different models for the same software functional requirements. To overcome this 
problem, we plan to demonstrate that for two or more different models of the same functional 
requirements, the generated test cases allow to detect the same bugs (Cf. Chapter 10 — Section 
8.B). 


VI. Conclusion 


Managing the software functional requirements is considered as one of the key issues in the 
software development process. In fact, these requirements are the main input for the design 
and implementation processes of the software product but also for the verification and 
validation processes. Ten years ago, formal methods were rarely used in automotive industry, 
contrary to medical, avionics and railways industries. Now, in automotive industry, semi- 
Jormal and formal methods are more and more used to specify software functional 
requirements (Cf. Diagnosis 3). However, there is a lack of a standard formalism shared 
between carmakers and suppliers. In fact, for each project, the supplier has to adapt its 
processes to the formalism used by the carmaker. 


In this chapter, we developed our new formal and simulation language to model software 
functional requirements (Cf. Diagnosis 6). À simulation model of these requirements could 
help to avoid ambiguity, incompleteness and inconsistency in customers” requirements (Cf. 
Diagnosis 5). Development and validation teams could communicate more easily with the 
customer and fix specification’s problems (Cf. Diagnosis 2). Moreover, through a simulation 
model, one could automate the assessment process of all the expected outputs values of a 
software product (Cf. Diagnosis 10, 12 and 15). In fact, when designing test cases, test 
engineers could perform the selected operation on the requirements model and assess the 
expected output values automatically by simulating the model. Finally, one could formally 
measure the coverage of the requirements model (Cf. Diagnosis 11). 


In the following chapter, we develop how a modeler can verify and validate the completeness, 
the consistency, the accuracy and the compliance of a requirements model with the 
carmaker’s requirements. 
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CHAPTER 6. VERIFICATION AND 
VALIDATION OF A SOFTWARE FUNCTIONAL 
REQUIREMENTS MODEL 
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I. Introduction 


Simulation models are increasingly being used in problem solving and in decision making. 
The developers and users of these models, the decision makers using information derived 
from the results of the models, and people affected by decisions based on such models are all 
rightly concerned with whether a model and its results are “correct”. This concern is 
addressed through Model Verification and Validation (Model V&V). In this dissertation, we 
adopt the definition of Model V&V proposed by Balci in (Balci 1997). 


Definition 6.1: Model Verification (Balci 1997) 


Model Verification is substantiating that the model is transformed from one form into 
another, as intended, with sufficient accuracy. Model verification deals with building the 
model right. The accuracy of transforming a problem formulation into a model specification 
or the accuracy of converting a model representation from a micro flowchart form into an 
executable computer program is evaluated in model verification. 


Definition 6.2: Model Validation (Balci 1997) 


Model Validation is substantiating that the model, within its domain of applicability, behaves 
with satisfactory accuracy consistent with the M&S (Modeling and Simulation) objectives. 
Model validation deals with building the right model. 


Model V&V are essential parts of the model development process if the model has to be used 
by organizations. Model V&V is not a phase or step in the M&S life cycle, but a continuous 
activity throughout the entire M&S life cycle. The M&S life cycle should not be interpreted as 
strictly sequential. The M&S life cycle is iterative in nature and reverse transitions are 
expected. Deficiencies identified by Model V&V activity may necessitate returning to an 
earlier process and starting all over again. The Model V&V activity throughout the entire M&S 
life cycle is intended to reveal any quality deficiencies that might be present as the M&S 
progresses from the problem definition to the completion of the M&S application. Errors 
should be detected as early as possible in the M&S life cycle. 


In this chapter, we develop scenarios in order to verify and validate a software functional 
requirements model developed using our formal simulation language. À survey on verifying 
and validating simulation model 1s performed in Section 2. We consider a simplified version 
of the modeling process. We discuss the basic approaches used in deciding model validity. 
We also describe various Model V&V techniques. Based on the literature review, techniques 
and rules to help modelers in validating the Conceptual Model, verifying the Computerized 
Model and finally checking the Operational Validity of a requirements model are respectively 
proposed in Section 3, 4 and 5. These proposals take the industrial constraints and the 
automotive context into account. 


IL. À survey on verifying and validating a simulation model 


A model should be developed for a specific purpose and its validity determined with respect 
to that purpose. If the purpose of a model is to answer a variety of questions, the validity of 
the model needs to be determined with respect to each question. Several sets of experimental 
conditions are usually required to define the domain of a model’s intended applicability. A 
model may be valid for one set of experimental conditions and invalid in another. À model is 
considered valid for a set of experimental conditions 1f its accuracy is within its acceptable 
range, which 1s the amount of accuracy required for the model’s intended purpose. Several 
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versions of a model are usually developed prior to obtaining a satisfactory valid model. The 
substantiation that a model is valid (e.g. Model V&V) is generally considered to be a process 
and is usually part of the model development process. 


It is often too costly and time consuming to determine that a model is absolutely valid over 
the complete domain of its intended applicability. Tests and evaluations are conducted until 
sufficient confidence 1s obtained that a model can be considered valid for its intended 
application. The relationships of cost (a similar relationship holds for the amount of time) of 
performing model validation and the value of the model to the user as a function of model 
confidence is proposed by Sargent in (Sargent 2005) and illustrated in Figure 6.1. The cost of 
model validation is usually quite significant, particularly when extremely high model 
confidence is required. 


Value Value 


Cost Model 
Cost to 


0% Model Confidence 100% 
Figure 6.1 —- Model confidence (Sargent 2005) 


A. A simplified version of the modeling process 


Sargent (Sargent 2005) has proposed a simplified version of the modeling process in Figure 
6.2. The Problem Entity 1s the system (real or proposed), idea, situation, policy, or phenomena 
to be modeled; the Conceptual Model is the mathematical/logical/verbal representation of the 
Problem Entity developed for a particular study; and the Computerized Model is the 
Conceptual Model implemented on a computer. The Conceptual Model is developed through 
an analysis and modeling phase, the Computerized Model is developed through a computer 
programming and implementation phase, and inferences about the Problem Entity are 
obtained by conducting computer experiments on the Computerized Model in the 


experimentation phase. 
Problem 
Entity LL 
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Figure 6.2 — A simplified version of the modeling process (Sargent 2005) 
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In the next three sections, we develop the three Model V&V activities (Conceptual Model 
Validity, Computerized Model Verification, Operational Validity) related to the simplified 
version of the modeling process presented in Figure 6.2. 


1. Conceptual Model Validity 


Conceptual Model Validity is determining that 1) the theories and assumptions underlying the 
Conceptual Model are correct, and 2) the model representation of the Problem Entity and the 
model’s structure, logic, and mathematical and causal relationships are “reasonable” for the 
intended purpose of the model. Next, each sub-model and the overall model must be 
evaluated to determine if they are reasonable and correct for the intended purpose of the 
model. This should include determining if the appropriate detail and aggregate relationships 
have been used for the model’s intended purpose, and if the appropriate structure, logic, and 
mathematical and causal relationships have been used. The primary validation techniques 
used for these evaluations are Face validity and Traces (Cf. Section 2.C for more details on 
these techniques). Face validity is based on experts’ evaluation of the Conceptual Model in 
order to determine 1f it 1s correct and reasonable for 1ts purpose (Problem Entity). This usually 
requires examining the flowchart or graphical model, or the set of model equations. The use 
of Traces is the tracking of entities through each sub-model and the overall model to 
determine 1f the logic is correct and if the necessary accuracy is maintained. If errors are 
found in the Conceptual Model, it must be revised and Conceptual Model Validity performed 
again. 


2: Computerized Model Verification 


Computerized Model Verification ensures that the computer programming and 
implementation of the Conceptual Model are correct. To help ensure that a correct computer 
program 1is obtained, program design and development procedures found in the field of 
software engineering should be used in developing and implementing the computer program. 
One should be aware that the type of computer language used affects the probability of having 
a correct program. The use of a special-purpose simulation language generally results in 
having fewer errors than if a general-purpose simulation language 1s used, and using a general 
purpose simulation language generally results in having fewer errors than if a general purpose 
higher order language 1s used. Not only does the use of simulation languages increase the 
probability of having a correct program, programming time is usually reduced significantly. 
After the computer program has been developed and implemented, the program must be 
tested for correctness. Main functions but also sub-functions must be tested to see if they are 
correct. It is necessary to be aware while checking the correctness of the computer model that 
errors may be caused by the Conceptual Model or the computer implementation. 


3. Operational Validity 


Operational Validity is concerned with determining that the model’s output behavior has the 
accuracy required for the model’s intended purpose over the domain of its intended 
applicability. This is where most of the validation and evaluation techniques take place. The 
Computerized Model is used in Operational Validity, and thus any deficiencies found may be 
due to an inadequate Conceptual Model, an improperly programmed or implemented 
Conceptual Model (e.g. due to programming errors or insufficient numerical accuracy), or due 
to invalid data. AII of the Model V&V techniques discussed in Section 2.C are applicable to 
Operational Validity. Which techniques to use must be decided by the model development 
team and other interested parties. The major attribute affecting Operational Validity 1s 
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whether the Problem Entity (or system) is observable, where observable means it is possible 
to collect data on the operational behavior of the program entity. 


Finally, Data Validity is defined as ensuring that the data necessary for model building, model 
evaluation and testing, and conducting the model experiments to solve the problem are 
adequate and correct. 


B. How to decide whether a simulation model is valid or not? 


According to Sargent (Sargent 2005), three basic approaches are used in deciding whether a 
simulation model is valid or invalid. Each of the approaches requires the model development 
team to conduct the Model V&V as part of the model development process: 


1. The most common approach is based on the model development team who has to 
make the decision whether the model is valid or not. This is a subjective decision 
based on the results of the various tests and evaluations conducted as part of the model 
development process. 

2. Another approach, often called “/ndependent Verification and Validation” (IV&V), 
uses a third (independent) party to decide whether the model is valid. The third party is 
independent of both the model development team and the model user(s). After the 
model is developed, the third party conducts an evaluation to determine its validity. 
Based upon this validation, the third party makes a subjective decision on the validity 
of the model. The evaluation performed in the /V&V approach ranges from simply 
reviewing the Model V&V conducted by the model development team to a complete 
verification and validation effort. According to Wood (Wood 1986), a complete 1V&V 
evaluation is extremely costly and time consuming for what is obtained. 

3. Balci (Balci 1989) proposes an approach based on a scoring model for determining 
whether a model is valid or not. Scores (or weights) are determined subjectively when 
conducting various aspects of the validation process and then combined to determine 
category scores and an overall score for the simulation model. À simulation model is 
considered valid if its overall and category scores are greater than some passing 
score(s). This approach is infrequently used in practice. Sargent (Sargent 2005) does 
not believe in the use of a scoring model for determining validity, because 1) the 
subjectiveness of this approach tends to be hidden and thus appears to be objective, 2) 
the passing scores must be decided in some (usually subjective) way, 3) a model may 
receive a passing score and yet have a defect that needs correction, and finally 4) the 
score(s) may cause overconfidence in a model or be used to argue that one model is 
better than another. 


Several versions of a model are usually developed in the modeling process prior to obtaining a 
satisfactory valid model. During each model iteration, Model V&V are performed. A variety 
of techniques could be used. In the next section, we develop these techniques. 


C. Model Verification and Validation techniques 


Taxonomy of more than 77 Model V&V techniques for simulation models is identified in 
(Balci 1997). Most of these techniques come from the software engineering discipline and the 
remaining are specific to the modeling and simulation field. Details on these techniques are 
proposed in (DoD 1996, Balci 1997). The taxonomy used by the authors classifies the Model 
V&V techniques into four primary categories: informal, static, dynamic, and formal. The use 
of mathematical and logic formalism by the techniques in each primary category increases 
from informal to formal. Likewise, the complexity also increases as the primary category 
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becomes more formal. In the following, we describe various techniques used in Model V&V. 
Most of the techniques described here are found in the literature (Balci 1984, Sargent 2005), 
although some may be described slightly differently (adapted to our context). They can be 
used either subjectively or objectively. By ‘“objectively”, we mean using some type of 
statistical test or mathematical procedure (e.g. confidence intervals). À combination of 
techniques 1s often used for validating and verifying the sub-models and the overall model. 


Animation: The model’s operational behavior is displayed graphically as the model moves 
through time. 


Comparison to Other Models: Various results (e.g. outputs) of the simulation model being 
validated are compared to results of other (valid) models. 


Degenerate Tests: The degeneracy of the model’s behavior is tested by appropriate selection 
of values of the input and internal parameters. 


Event Validity: The events of occurrences of the simulation model are compared to those of 
the real system to determine 1f they are similar. 


Extreme Condition Tests: The model structure and output should be plausible for any extreme 
and unlikely combination of levels of factors in the system. 


Face Validity: Face validity is asking people knowledgeable about the system whether the 
model and/or its behavior are reasonable. This technique can be used in determining if the 
logic in the Conceptual Model is correct and if a model’s input-output relationships are 
reasonable. 


Fixed Values: Fixed values (e.g., constants) are used for various model input and internal 
variables and parameters. This should allow the checking of model results against easily 
calculated values. 


Historical Data Validation: If historical data exist (or if data are collected on a system for 
building or testing the model), part of the data is used to build the model and the remaining 
data are used to determine (test) whether the model behaves as the system does. 


Internal Validity: Several replications (runs) of a stochastic model are made to determine the 
amount of (internal) stochastic variability in the model. À high amount of variability (lack of 
consistency) may cause the model’s results to be questionable and may question the 
appropriateness of the system being investigated. 


Parameter Variability - Sensitivity Analysis: This technique consists of changing the values of 
the input and internal parameters of a model to determine the effect upon the model’s 
behavior and its output. The same relationships should occur in the model as in the real 
system. 


Predictive Validation: The model is used to predict (forecast) the system behavior, and then 
comparisons are made between the system’s behavior and the model’s forecast to determine if 
they are the same. The system data may come from an operational system or from 
experiments performed on the system. 


Traces: The behavior of different types of specific entities in the model is traced (followed) 
through the model to determine 1f the model’s logic is correct and if the necessary accuracy 1s 
obtained. 


Turing Tests: People who are knowledgeable about the operations of a system are asked if 
they can discriminate between system and model outputs. 
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Unfortunately, no algorithms or procedures exist to select which techniques to use. However, 
some attributes that affect which techniques to use are discussed by Sargent in (Sargent 1984). 
In the next three sections (Section 3, 4 and 5), we specify techniques, rules and scenarios to 
help modelers in validating the Conceptual Model, verifying the Computerized Model and 
finally checking the Operational Validity of a requirements model. Our proposals take not 
only the Sargent’s recommendations into account but also our industrial context. 


III. Using the experts’ knowledge to validate a Conceptual requirements 
Model (Conceptual validity) 


In our context, the Conceptual Model is developed through an analysis and modeling of the 
software functional requirements. For each software functionality, modelers draw a sketch of 
the requirements model by (Cf. Chapter 5 — Section 3): 


1. identifying the input and output signals and their domains, 

2. grouping the functional requirements according to their types (combinatorial or 
sequential), 

3. identifying the elements (DT and FSM) and the intermediate signals and their domains 
and 

4. finally specifying each element. For a DT, identify the conditions and their associate 
actions. For a FSM, identify the strates and their associate actions, the transitions and 
their associate conditions and if needed the internal and timing signals. 


Once the Conceptual Model is designed, each element and the overall model must be 
evaluated to determine if they are reasonable, correct and complete regarding the carmaker’s 
requirements. We propose to use Face validity and Turing tests in order to valid our 
Conceptual Model. In fact, the experts’ knowledge is the main source of validating our 
Conceptual Model. People knowledgeable about the system under test are asked to 
discriminate between the model and the carmaker’s requirements and to give their confidence 
in the model and/or its behavior. 


IV. A set of integrity rules to verify a Computerized requirements Model 


The Computerized Model is developed through a computer programming and implementation 
of the Conceptual Model. We provide modelers a high level graphical language to help them 
computerizing their Conceptual requirements Models. The main items of this language are 
illustrated in Figure 6.3. 
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Functionality 
—Ck}, Clock 
Config Configuration signal 
ni Input signal 
_O ; Output signal 
Int, Intermediate signal 
DT 
EU Decision Table element 


Finite State Machine element 


vd 
+ © 


State 
Transitions 
iitemal Signal FSMinternal signal 
LA FSM timing signal 
Figure 6.3 — A high level graphical language to computerize a Conceptual requirements 
Model 


Computerized Model Verification ensures that the computer programming and 
implementation of the Conceptual Model are correct. The use of a graphical simulation 
language generally results in having fewer errors and programming time 1s usually reduced 
significantly. To help ensure that a correct computer model 1s obtained, we develop a set of 
integrity rules to be checked automatically on the computer model. These rules are developed 
in Table 6.1. 
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| Rule# | Rule description 


Rule1 Each functionality musthave one clock 


Rule2 Each functionality musthave atleast 1 inputand 1 outputsignals 


Each functionality must have at least one element (a Decision Table or a Finite State 


RULES Machine) 


Rule4  Altheinputsignals of the functionality must be inputs of elements 
Rule5  Allthe outputsignals of the functionality mustbe outputs of elements 


Rule6  Altheintermediate signals of the functionality must be inputs or outputs of elements 


All the inputs and outputs of elements must be input, output or intermediate signals of 
the functionality 

Each value of an input, output and intermediate signal of the functionality must be 
considered in atleast one condition or action of an element 


Rule 7 
Rule 8 


Rule9 Each DT musthave at least one condition 


Rule 10 Each condition of a DT musthave one associated action 


Each condition of a DT must have at least one input or intermediate signal of the 
functionality 
Each action of a DT must have at least one output or intermediate signal of the 
functionality 


Rule 11 
Rule 12 
Rule 13 Each FSM musthave at least two states and two transitions 
Rule 14 Each transition of a FSM must have at least one condition 


Rule 15 Each state of a FSM must have one associated action 


Each condition of a FSM must have at least one input or intermediate signal of the 
functionality 

Each action of a FSM must have at least one output or intermediate signal of the 
functionality 

Each state of a FSM must have at least one transition that gets in the state and one 
transition that gets out ofthe state 


Rule 16 
Rule 17 
Rule 18 


Rule 19 Each transition of a FSM must have an origin and a destination state 


Table 6.1 — Integrity rules for verifying a Computerized requirements Model 


V. Three possible scenarios to validate a Computerized requirements 
Model (Operational Validity) 


Computerized Model Verification ensures that mistakes have not been made in the computer 
implementation of the Conceptual Model. It does not ensure the compliance of the computer 
model with the (original) carmaker requirements. The Operational Validity aims to verify that 
the computer model behavior has the accuracy required by the carmaker. To do this, computer 
experiments are conducted on the Computerized Model in the experimentation phase. This is 
where most of the model deficiencies are detected. Errors may be due to an erroneous 
Conceptual Model or to programming errors in computerizing the Conceptual Model. In our 
context, all of the Model V&V techniques discussed in Section 2.C are applicable. Which 
techniques to use must be decided by the constraints on the system under test but also by the 
model development team. In fact, we identify three possible scenarios to help modelers 
validating a Computerized requirements Model (Operational Validity). These scenarios can be 
used concurrently or separately. 
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A. First scenario: Animate our requirements model 


The most used technique to validate a simulation model 1s obviously to animate and trace it 
(Cf. Figure 6.4). The behavior of intermediate and output signals is provided graphically 
through time. However, two questions can be raised when animating a model 1) How do I 
choose the input data of the model? and 2) How can I be sure that the model behaves well 
(expected output data)? In order to answer the first question, we can refer to some of the 
Model V&V techniques provided in Section 2.C (for instance, degenerate tests, extreme 
condition tests, fixed values, and parameter variability - sensitivity analysis). The second 
question 1s more related to the formalism of the (original) carmaker requirements. In case of 
informal or semi-formal requirements, modelers have to predict the system behavior by 
analyzing the requirements. In case of formal and simulated requirements, expected output 
data can be assessed automatically by simulating the requirements. 


Modifications 


Model Intermediate : 


. Input data, 7 


é.! : and Output data 
We 7 Len (by simulation) 
Engineer = À ! No 
o 
G ! Y. 
e ------>x'Equal?>--- ES 
G : 
og} 
. . Ÿ 
Roques enAiVsRs Expected Intermediate 
: on ! . and Output data 
Requirements simulation 
Figure 6.4 - Animate the requirements model 
B. Second scenario: Simulate the test cases delivered by the carmaker on 


our requirements model 


Sometimes, carmakers deliver a set of test cases for the software product under test. These 
test cases can be used to valid our requirements model (Cf. Figure 6.5). 


Quality of the design of test cases for automotive software: design platform and testing process 


175 


Verification and validation of a software functional requirements model R. AWEDIKIAN 


Modifications 
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Figure 6.5 — Simulate the test cases delivered by the carmaker on our requirements 
model 


C. Third scenario: Compare our requirements model to another valid model 
of the requirements 


Ten years ago, simulation methods were rarely used in automotive industry. Now, automotive 
electronics architecture becomes more and more complex and carmakers outsource the design 
of electronic products. Therefore, it becomes crucial for carmakers to simulate their global 
electronics architecture in order to better integrate and validate different electronic parts from 
different suppliers. Presently, in automotive industry, formal (simulation) methods are more 
and more used to specify software functional requirements (Cf. Chapter 2 — Section 4.A). In 
fact, simulation helps engineers to make better decisions all along the product life cycle and 
detect problems early in the development process. Unfortunately, there is a lack of a standard 
Jormalism shared between carmakers and suppliers (Cf. Diagnosis 3). However, some 
existing simulation tools (StateMate*?, Matlab/Simulink**) are currently used by carmakers 
and suppliers to simulate software specifications. A graphical interface generated 
automatically from a formal specification (delivered by a carmaker to Johnson Controls) of 
the “Front Wiper” functionality 1s 1llustrated in Figure 5.6. An engineer can animate this 
model manually or simulate a set of input data automatically (set values on the input signals) 
and check the expected behavior of the functionality (check values on the output signals). 


* http:/www.telelogic.com/products/statemate/index.cfm (Consulted on November 2008) 
 http:/www.mathworks.com/products/simulink/ (Consulted on November 2008) 
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Figure 6.6 — A graphical interface generated automatically from a formal specification 
of the “Front Wiper” software functionality 


Once a simulation model of the software functionality under test exists (Cf. Figure 6.7), an 
engineer can choose a set of input data (using techniques such as degenerate tests, extreme 
condition tests, fixed values, parameter variability - sensitivity analysis) and simulate these 
data on the “valid” model (model delivered by the carmaker) and on our requirements model 
in order to verify the validity of our model. 
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Figure 6.7 —- Compare our requirements model to another valid model of the 
requirements 


VI. Conclusion 


Assessing credibility of a simulation model is an onerous task. Applying Model V&V 
techniques throughout a simulation model is time consuming and costly. However, the Model 
V&V activity 1s extremely important for successful completion of complex and large-scale 
Modeling and Simulation (M&S) efforts. Unfortunately, there is no set of specific tests that 
can easily be applied to determine the “correctness” of the model. Furthermore, no algorithm 
exists to determine what techniques or procedures to use. Every new simulation project 
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presents a new and unique challenge. However, there is considerable literature on Model 
V&V. 


In this chapter, we developed a process to verify and validate a software functional 
requirements model. We use proposals developed in the literature as a starting point for 
defining methods and techniques more adapted to our context. We consider a simplified 
version of the modeling process: Problem Entity, Conceptual Model and Computerized 
Model. Firstly, we propose to validate (Conceptual Validity) a Conceptual requirements 
Model via experts’ knowledge. Secondly, we define a set of integrity rules to verify a 
Computerized requirements Model. Finally, we develop three possible scenarios to validate 
(Operational Validity) a Computerized requirements Model. 


In the following chapter, we describe how test cases can be generated automatically from a 
requirements model. 
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CHAPTER 7. AUTOMATIC GENERATION OF 
TEST CASES 
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I. Introduction 


As the software products become more and more complex (Cf. Chapter 1), it 1s 1llusory to be 
able to check that the software product responds correctly to all possible operations. In 
Chapter 8 — Section 2, we further demonstrate that soffware testing is a NP-Complete problem 
and therefore it is impossible to be able to cover all the operation space. In Johnson Controls, 
the current strategy to select operations Within the operation space (operation selection 
strategy) is a manual subjective one based the test engineers’ experience and intuition (Cf. 
Diagnosis 13). After choosing an operation to be performed on the software under test, test 
engineers analyze the source code and/or the carmaker requirements of this software in order 
to assess the expected values to be checked on some output signals. In fact, this assessment 1s 
based on the engineers’ understanding of the code and/or requirements and may lead to errors 
(Cf. Diagnosis 12). In automotive industry, these tasks become laborious tasks that account 
for more than 50% of the total time and budget of a software project. In fact, the stopping 
criteria used is based on fest coverage. Researches in code coverage measurement have 
reached a high level of maturity and many automated tools were commercialized (Cf. Chapter 
2 — Section 6.A.1). However, requirement coverage is still immature and the accuracy of a 
requirement coverage measurement depends on the degree of formalism used when 
specifying a set of requirements (Cf. Diagnosis 11). In addition, sometimes, for rime and 
budget reasons, test engineers stop designing fest cases even if 100% coverage is not reached. 


In this chapter, we develop our strategy to generate fest cases (operations and expected 
outputs) automatically from the requirements model. We also describe our stopping 
aggregated criterion based on formal measurement of coverage. The test case generation 1s 
based on a new concept named “operation matrix” presented in Section 2. The process of 
generating a test case 1s described in Section 3. The quality objectives and the time and cost 
constraints when designing fest cases are developed in Section 4. À new stopping aggregated 
criterion 1s proposed in Section 5. Finally, our heuristic algorithm to optimize the generation 
of test cases 1s Specified in Section 6. 


IT. The new concept of ‘‘operation matrix” 


The generation of operations and inter-operation times for à test case is performed based on 
the concept of “operation matrix”. In fact, for each software functionality under test, we 
propose to set probabilities and time intervals between all possible successive operations. 
Therefore, we build a matrix that we name “operation matrix” which is a square matrix with 
all possible operations in columns and in rows. Between the two operations of a pair we 
define: 


e The probability that the two operations be in sequence. The total of the probabilities 
by row must be equal to 1. 

e The inter-operation time between these two operations, modeled as an interval of 
possible values (a uniform probability). 


Let us consider a software functionality with 3 input signals and two output signals: II, 
Domain = {0, 1}; 22, Domain = {1, 2, 3}; 13, Domain = {0, 1}; O1, Domain = {0, 1}; O2, 
Domain = {0, 1}. The “operation matrix” associated to this example is illustrated in Figure 
Va 
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Moreover and through the “operation matrix”, engineers can enrich the requirements model 
with knowledge on the end-user (driver) recurrent operations and the test engineers’ 
experience. Indeed, high probabilities and specific inter-operation times can be set between 
recurrent and/or critical operations. The use of driver behavior’s profile, past bugs and 
existing test cases in order to refine the operation space description is developed in Chapter 
8. 


One major question is: How an engineer can design an “operation matrix”? The basic solution 
is to fill in manually each case of the “operation matrix” by a succession probability and a 
üme interval. However, a functionality can have more than 20 input signals and 100 possible 
values for these signals. In fact, the domain length of an analog input signal (for instance, 
vehicle speed) depends on the level of details when sampling the analog domain. In 
consequence, the size of an “operation matrix” can easily reach the 10000 cases which 
become inconceivable to be filled manually. One solution is to generate the “operation 
matrix” automatically. For each functionality under test, we propose to generate two 
“operation matrices” automatically. For the two matrices, the time interval can be set 
automatically to a “generic” interval defined by experts. Let us consider the same example of 
the Figure 7.1. The first matrix called “Nominal 1” (Cf. Figure 7.2) considers that all the 
operations have the same succession probability. 


Defined by experts 


{Succession probability ; Time interval} 


0.14 + 0.14 + 0.14 + 0.14 + 0.14 + 0.14 + 0.14=1 


Figure 7.2 - An example of a Nominal 1 “operation matrix” 
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The second matrix called “Nominal 2” (Cf. Figure 7.3) considers that the probability to 
choose an operation on the input signal I; is the same than the one on the input signal I.. 


Defined by experts 


{0, 
{0,1 


}1{0,17;IX; 
0,11:/X;Y]} | {0,17:[X;V]} | 40,17: IX ;V]} 
0,11:/X;Y]} | {0,17:[X;V]} | 40,17: IX ;V]} 


{Succession probability ; Fime interval} 


0.17+0.17 = 0.33 0.11+0.11+0.11 = 0.33 0.17+0.17 = 0.33 


0.33+0.33+0.33 = 1 
Figure 7.3 - An example of a Nominal 2 “operation matrix” 
Moreover, once these matrices are generated automatically, engineers have the possibility to 
adjust manually the succession probability and the time interval between some specific 
operations. Following the engineers’ modifications, the probability distribution by row is 


updated in order to take into account the matrix constraints. For instance, let us consider the 
Nominal 1 “operation matrix” of the Figure 7.2. One engineer can decide to: 


e set the succession probability between the operation “I1=0” and the operation 
“I=1" to 0.8 


e and set all the time intervals after the operation “13=0” to [X1, Y1] 


The modified Nominal 1 “operation matrix” 1s presented in Figure 7.4. 


Quality of the design of test cases for automotive software: design platform and testing process 


182 


Automatic generation of test cases R. AWEDIKIAN 


polie lslete. 


ao 

(=, 

ü 

= 

[e] 

Q 
LOT 10,033:/X:v)) | 10,8:/X:V]) À {0,033:/X :Y) 0,033: [X :Y) 0,033: IX :Y) 0,033: [X :Y] 0,033: [X :Y] 
ERREURS RU (14: POV} | 10,14: IX} XVI} | 40,14:D0:v7 | (0,14: 


CO ANT | 
(0,14: XVI} Pen | (O4: IX | (0.14: | (0.14fD VI | {0.14:IX:V)) | (0.14:IX:V) 
(0,14: XVI} jora:priv | to14:x:vn | 1014 | {0.14:xv) | (0.14:px:V) 


2 : 
O 40,14; X1 ;Y#]} | (0,14; [X1 ;Y1]} ? 0,14; [X1 ;Y1]} | {0,14; EX1 ;Y1]} | {0,14; [X1 ;Y1]} | {0, 14; [X1 ;Y1]} 
AVE ; IX ; 


otaipovn | on | Ca | of | wine | tor:x mn 


3 
Ü 
3 
Q 
+ 


0.8 0.033+ 0.033+ 0.033+ 0.033+ 0.033+ 0.033 = 0.2 
0.8+0.2 = 1 
Figure 7.4 - An example of a Nominal 1 “operation matrix” after engineers’ 
modifications 
III. How to generate a “Test Case’’? 


Generating a fest case automatically requires generating a set of fest steps until stopping 
criterion 1s Validated. The process of generating a test case is illustrated in Figure 7.5. 


Generate a 
Test Step 
Stopping 
Criterion 


Yes 


Figure 7.5 — The process of generating a test case 


The definition of our stopping aggregated criterion is developed in Section 5. In the 
following, we focus on the generation of a test step. Based on the definition of a fest step (Cf. 
Definition 2.11), designing a test step requires to choose an operation, an inter-operation time 
and assess the expected results to be checked on the output signals of the software under test. 
Through our approach, two automated activities are necessary to generate a fest step: 


A. Activity 1: À Monte Carlo simulation on the “operation matrix” 


In order to generate an operation and an inter-operation time, We propose to perform a Monte 
Carlo simulation on the “operation matrix”. Two steps are required: 
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e Step 1: an operation is chosen according to the probabilities on successive operations. 
In software testing (Marre 1992), this technique is known under the sfatistical testing 
technique. Before start generating a test case, the input signals of the requirements 
model are set to specific values. Therefore, the starting operation of the test case is 
randomly chosen among the initial operations (initial values of the input signals). 
Sometimes, the starting operation 1s chosen in order to favor a specific succession of 
operations at the beginning of the test case. 

e Step 2: the inter-operation time is randomly chosen within the time interval of the 
chosen operation. 


B. Activity 2: À simulation of the software requirements model 


The chosen operation is performed on the requirements model and a simulation of the model 
(synchronized by the cycle time of the Clock signal) starts until the inter-operation time ran 
out. The values on the output signals of the model are the expected results of the test step. Let 
us consider the example of Figure 7.1 with the “operation matrix” of Figure 7.4. The process 
of generating a test step 1s illustrated using this example in Figure 7.6. 
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Test Step No Test Actions Expected Results 


4 M=1 O1=0 
Wait 250 ms O2= 


0,033 0,8 0,033 | 0,033 | 0,033 | 0,033 | 0,033 
14 1B8LL200.400 | 1200,400j | 1200.400] | 200.400! | 200.400] | 1200.400] | 1200.400 
EVE 0,14 0,14 0,14 0,14 0,14 0,14 0,14 
Activity 1 200,400] | [200,400] | [200.400] | [200.400] | [200,400] | [200.400] | [200,400 
0,14 0,14 0,14 0,14 0,14 0,14 0,14 
Monte Carlo 200.400] | [200.400] | 1200.40] | 120,400] | [200.400] | 120,400] | 200.400) 
; ; 0,14 0,14 0,14 0,14 0,14 0,14 0,14 
simulation on an 200.400] | [200.400] | 1200.40] | 200,400] | [200.400] | 200,400] | 200.400) 
r : sn 0,14 0,14 0,14 0,14 0,14 0,14 0,14 
operation matrix 200,400] | [200.400] | 1200.40] | 200,400] | [200.400] | 200,400] | [200.400 
0,14 0,14 0,14 0,14 0,14 0,14 0,14 
600,900] | [600,900] | [600,900] | 100.900] | [600.900] | 600,900] | 160,900; 
0,14 0.14 0,14 0,14 0,14 0,14 0,14 
200,400] | [200.400 | 120.400] | 200.400] | 120,400) | 200.400] | 1200,400 
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0,033 GE 0038 | 0033 | 0,039 | 0038 | 0.033 
it 200,400] | [200,400] | [200,400] | [200,400] | [200,400] | [200,400] | [200,400] 
0,14 0,14 0,14 0,14 0,14 0,14 0,14 
200,400] | [200,400] | [200,400] | [200,400] | [200,400] | [200,400] | [200,400 

0,14 0,14 0,14 0,14 0,14 0,14 0,14 
[200,400] | [200,400] | [200,400] | [200,400] | [200,400] | [200,400] | [200,400 

12 0,14 0,14 0,14 0,14 0,14 0,14 
[200,400 [200,400] | [200,400] | [200,400] | [200,400] | [200,400 
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Choosethe nextoperation 
according to the probabilities 


“H=1"is the last (generationofthe nexttest step) 


chosenoperation 
Activity 2 

. : 411 014 0,14 0,14 0,14 0,14 0,14 0,14 
Simulation of the 200,400] | [200,400] | [200,400] | [200,400] | [200,400] | [200,400] | [200.400 

: 0,14 0,14 0,14 0,14 0,14 0,14 0,14 
requirements 2] 1200.400) | 120,400] | 200.400] | 200,400] | [200.400] | 200.400] | 200,400 

0,14 0,14 0,14 0,14 0,14 0,14 0,14 
model 3] 120,400] | 200,400] | 1200400] | [200,400] | 120.400] | [200.400] | 1200400: 

ol. 014 0,14 0,14 0,14 0,14 0,14 0,14 
600,900] | 160,900] | 1600,900} | 1600.00] | 160,900] | 1600.900] | 100,900 

11. 014 0,14 0,14 0,14 0,14 0,14 0,14 
200,400] | 200,400] | 1200,400} | [200,400] | [200,400] | [200,400] | 200,400 


Figure 7.6 — The process of generating a test step 
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IV. Test generation objectives and constraints 


Testing software exhaustively remains a major problem from the computing point of view. 
Therefore, software testing must often be based on specific assumptions and objectives which 
help test engineers and managers to decide when to stop the testing protocol. In order to 
monitor our automatic generation of fest cases, We develop an objective function based on a 
Jormal structural (code) and functional (requirement specification) coverage and the 
execution time and cost of generated tests. In our approach, test engineers can generate fest 
cases according to their quality objectives but also time and cost constraints. 


A. Structural (code) coverage objectives 


While generating a test case, and for each generation of a fest step, we execute the fest step on 
the software product under test and we evaluate the code coverage in terms of statements, 
procedures, conditions and decisions coverage. To do so, we use C-Cover from Bullseye as a 
code coverage measurement tool. À detailed description of the code coverage is given in 
Chapter 2 — Section 6.A.1. Since the code coverage measurement is already formalized using 
commercial tools, we focus our efforts on formalizing the requirement coverage 
measurement. 


B. Functional (requirement specification) coverage objectives 


Once we define a model to formally represent and simulate the software functional 
requirements, we consider a formal coverage rate of the requirements model. In Figure 7.7, 
we illustrate the functional coverage indicators through the example of Figure 5.3. 


Finite State Machines coverage 


S 


# ".. Decision Tables coverage 


s 


Element NA 
qu Element N3 ( 
pl one Decision Element N4 
Table Finite 
à State 
Machine 


Element N?2 
Decision 
Table 


Operation matrix coverage 


Signals domain coverage 
Figure 7.7 - Functional (requirement specification) coverage indicators 


While generating a fest case, test engineers can visualize in real time the covered zones of the 
requirements model (Cf. Figure 7.8, 7.9, 7.10 and 7.11). 
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1. The coverage rate of a signal domain 


Each input, output or intermediate signal has a discrete domain. The signal domain coverage 
of a requirements model consists of the coverage rate of the domains of these signals. In 
addition, since testing the boundary values of a signal often reveals many problems, we also 
assess the coverage rate of the minimum and maximum values of each signal. In Figure 7.6, 
we illustrate the coverage of a signal by a practical example. After generating a test case, 
some values of the signals domains have been highlighted. In fact, the signal “Signal_3” 1s 
covered at 100% while the two values of this signal were visited at least once by the generated 
test case. The signal “Signal_1” has a coverage rate of 33,33% (1 value visited over 3 values 
in total). 


Signal name Signal domain, 
Hëk—— Signalcoveredat33,33% 


<——Signalcoveredat 100% 


Figure 7.8 — Signals domain coverage 
2. The coverage rate of an operation matrix 


The operation matrix coverage of a requirements model consists of the coverage rate of all 
successions between pairs of operations visited. Once a succession probability is set between 
each two operations, We define a coverage rate of the critical successions where the 
succession probability 1s above a certain level defined by the engineer. In Figure 7.9, we 
illustrate the coverage of an “operation matrix” by a practical example. After generating a 
test case, some cases of the matrix have been highlighted. In fact, in the generated fest case, 
the operation #4 has followed the operation #1, the operation #2 has followed the operation 
#2, the operation #2 has followed the operation #3 and so on. This way, we compute the 
coverage rate of successions between pairs of operations (around 38%; 5 successions of 
operations Were covered over 13 possible successions) 


Operations Op1 Op2 Op3 Op4 | Covered succession 
LL . 
—— 
Q 
Op4 0,2-[100:400] [o2-fo0200 | 0 | Non-covered succession 
of operations 
Figure 7.9 —- Operation matrix coverage 
3. The coverage rate of an element (DT or FSM) 


The element coverage of a requirements model consists of the coverage rate of the conditions 
of each Decision Table (DT) and the coverage rate of the sfates, transitions and conditions of 
each Finite State Machine (FSM). In Figure 7.10, we illustrate the coverage of a DT by à 
practical example. After generating a test case, some conditions of the DT have been 
highlighted. In fact, the conditions are covered at 75% (3 conditions visited over 4 conditions 
in total). 
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Conditions Actions 
Covered condition—+ — 


Non-covered condition—p""= 5 "17" 20 25 
Figure 7.10 — Decision Table coverage 


In Figure 7.11, we illustrate the coverage of a FSM by a practical example. After generating a 
test case, some states and transitions of the FSM have been highlighted. In fact, the sfates are 
covered at 75% (3 states visited over 4 states in total). The transitions are covered at 43% (3 
transitions visited -different from 0%- over 7 transitions in total). The conditions are covered 
at 29% ((50%+0%+100%+0%+50%+0%+0%)/7=29%). 


50%ofthe conditions are covered 
=> Covered transition 


Coveredstate 


None ofthe conditions is covered (0%) 
Non-covered state —| State4 | => Non-coverediransition 


Figure 7.11 — Finite State Machine coverage 


Moreover, when designing the requirements model, engineers can affect to conditions, states 
and transitions a normalized criticity level between 0 and 1. Consequently, we define a 
second set of coverage rates for expressing the degrees of coverage of the most critical 
conditions, states and transitions (in fact, this 1s a weighted coverage of an element). 


C. Test execution time and cost constraints 


Presently, in the automotive industry, the time and money spent to test a software product is 
the major criterion to stop testing. We have time and money spent to analyze carmakers” 
requirements, to design fest cases and to execute fesf cases on the software product under test 
(Cf. Chapter 2 — Section 6). In our approach, we generate test cases automatically and 
therefore, one can have a tendency to generate too many tests. However, executing fest cases 
on the software product under test and analyzing the results can cost too much time and 
money (Cf. Chapter 4 — Section 4.C.1) and more especially when the execution is performed 
manually by a test engineer (Cf. Chapter 2 — Section 5.D.1). In our approach, when 
generating a fest case, test engineers can set targets not to be exceeded (constraints) on time 
and cost indicators: 


e  Indicator 1: Execution time. The time that a test engineer will spend in executing 
manually the generated test case on the software product. 
e _Indicator 2: Number of test steps in the generated test case. 
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e _Indicator 3: Number of “distinct” test steps in the generated test case. Two test steps 
are distinct if they perform different operations. 


Let us consider the test case of the Figure 7.12. The total execution time is 1150 ms 
(250ms+200ms+300ms+400ms=1150ms, around 19 seconds). The rest case is composed 
from 4 test steps and the number of “distinct” rest steps is 3 (Test Step 1 and Test Step 4 
perform the same operation “I1=1”). 


Test Step No Test Actions Expected Results 
4 M = 1 O1=-0 
Wait 250 ms O2=1 
9 12=2 O1=0 
Wait 200 ms O2=1 
3 13= 1 O1=1 
Wait 300 ms O2=1 
à I =1 O1=0 
Wait 250 ms O2=1 


Figure 7.12 — An example of test case 


Constraints on time and cost are helpful in case of tight planning and budget on the project. It 
can also be useful on projects where the test execution 1s performed manually. In that case, 
the execution time and number of test steps must be reduced and the repetitive test steps or 
succession of test steps must be avoided. Typically, when testing a Graphical User Interface 
(GUT), test engineers have to check visually the expected results. Nevertheless, new testing 
platforms allow even to automate the testing of GUI using a camera system. 


V. Our stopping aggregated criterion 


A. The objective function combining objectives and constraints 


Based on the coverage objectives and the time and cost constraints identified in Section 4, we 
develop a panel interface to allow the test engineers to set precise targets on the test 
generation objectives and constraints (Cf. Figure 7.13). The quality objectives (code and 
requirement specification coverage) are expressed in terms of ratios of coverage and, then, 
are normalized which aim to reach a value of 100%. The execution time constraint 1s 
expressed in millisecond (ms). In addition, we define a set of weights (wi) that test engineers 
can associate for each defined target: 0 (to be ignored), 1 (not very important), 5 (important), 
10 (very important). The panel presented in Figure 7.13 helps test engineers to express their 
targets in terms of the required software quality and tests cost and therefore generate test 
cases fulfilling their objectives and constraints. 
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[ Targets Î \Weights 
Structural (code) Coverage Objectives 


Code statements Coverage % [o [o 
Code procedures Coverage # [o [o 


Code conditions Coverage CS, CONNNE 
Code decisions Coverage % CE CE 
- Functional (specification] Coverage Dbjectives 
+ Signals domains coverage 

Inputs domains Coverage 4 CE CE 
Outputs domains Coverage 20 pp 
Intermediates domains Coverage 0 
Inputs boundaries Coverage CO CNE 
Outputs boundaries Coverage AS | CNE 
Intermediates boundaries Coverage CAT CES 
Operation matrix coverage - 

Successive 2-Operations Coverage 20 KE 
Critical successive 2-Operations Coverage % CE CE 

£ Elements coverage 

DT Condition Coverage OS CON 
FSM State Coverage C2 à CEE 
FSM Transition Coverage 0 
FSM Condition Coverage C2 CR CRE 
OT Critical Condition Coverage CS CON 
FSM Critical State Coverage C2 CE 
FSM Critical Transition Coverage SSSR 
FSM Critical Condition Coverage CS “ONE 


Test ExecutionTime and Cost Constraints 
Test Case ExecutionTime (x1] ms F08000 CCS 
Test Step Number 0 
E—_—_—_—— 


Distinct Test Step Number 


Figure 7.13 — Panel of the quality, time and cost indicators for monitoring the automatic 
generation of test cases 


In fact, through our approach, the automatic generation of tests is monitored by the set of 
targets and weights predefined for each of the quality, time and cost indicators. During one 
test generation session, the targets may be completed following different orders and the first 
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target completed does not immediately stop the process. We stop only when the aggregated 
preference, F, defined as: 


F F > O, arg ef : (BTE X 2. Da C arg ef = Co X É 


| Î 
F 


constraints 


F 


objectives 


where Os are the coverage objectives, Cs are the normalized time and cost constraints and w:s 
are weights, attains zero or does not decrease for a certain number of successive generated test 
steps. This number is one of 8 parameters of the heuristic algorithm used to optimize the 
objective function (F) when generating a test case. The algorithm and its parameters are 
described in Section 6. Since the objectives Os are normalized to 100% and in order to have a 
consistent aggregated preference (F), we normalize to 100% the time and cost constraints 
(test case execution time, test step number, distinct test step number). These constraints are 
expressed in millisecond (ms) and in number of generated test steps. We illustrate our 
normalization process of these constraints through an example. At each time, test engineers 
decide to set a constraint ci, the normalized target of this constraint Crage{ci) 1S immediately 
set to 100%. For instance, once a test engineer decide to generate a fest case that the total 
execution time do not exceed 108000 ms (value set in the panel of quality, time and cost 
indicators, Cf. Figure 7.13), the normalized target of the test execution time constraint 1s set to 
100% (Crarge(test execution time)= 100%). After generating a set of fest steps, the normalized 
current value of this constraint (CcCurren(ci)) is assessed by calculating the ratio 
(current_constraint_value*100/target_constraint_value). In our example, we generate a set of 
test steps With a total execution time of 21600 ms. Therefore, Ccyren(test execution time) is 
assessed to (21600*100)/108000 (Ccwrenttest execution time)= 20%). 


B. A simple example to 1llustrate our stopping aggregated criterion 


Let us consider a practical software testing problem in order to illustrate the purpose of our 
objective function. Through the experience feedback of the software testing experts, some 
software bugs often occur when a signal 1s set to its boundaries values. Consequently, test 
engineers could always decide to generate a test case (a set of test steps) which aims to detect 
potential bugs related to the boundaries values. Hereafter, we consider the functionality which 
consists in managing the front wiper in a vehicle. The corresponding software component is 
made of 1229 Lines Of Code (blank and comment lines excluded), 18 input signals and 8 
output signals. We decide to generate a test case fulfilling the following targets and weights 
in terms of coverage objectives (Cf. Figure 7.13): 


- Cover the boundaries input signals at 85% with a weight of 5 
- Cover the boundaries output signals at 85% with a weight of 5 
- Cover the boundaries intermediate signals at 85% with a weight of 5 


While respecting the constraint: 
- Do not exceed 30 minutes (108000 ms) of tests execution with a weight of 10 


After generating a fest case With the objectives and constraints defined below, a test report is 
generated automatically. An excerpt of this report is illustrated in Figure 7.14. In this report, 
the reached (current) values on the objective and constraint indicators are illustrated. In fact, 
even if the inputs and outputs boundaries coverage have respectively reached and exceeded 
their targets (respectively 85% and 94% of coverage), our optimization algorithm did not stop 
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the generation of test steps expecting that the intermediate boundaries coverage reaches its 
target. But once the maximum fest execution time which has a weight of 10 (very important) 
has been exceeded (110255 ms instead of 108000 ms), the optimization algorithm decides to 
stop generating fest steps even if the intermediates boundaries coverage is not already 
reached. The excess of a constraint out of its bounds 1s accounted for a penalty that 
irremediably increases the overall objective value. 


Test Case Indicators Progress Current Target Weight 


Functional (specification) coverage objectives 


DT Condition [|] 68% 0 % 


0 

FSM State [SSs | 63% 0% 0 
FSM Transition [mn | 27% 0% 0 
FSM Condition [SSs | 30% 0% 0 


Inputs domains D | 53 % 0 % 
Outputs domains (SSSSSSSMM | 54% 0% 


(n] 
(n] 
Intermediates domains (s] 
5 
5 


Successive 2-Operations [a | 5% 0 % 0 


Test execution time and cost constraints 
[TE Execution TIME MINT TT (255 105000 10 | 
Test Step Number [MMM 445 Ô Ô 
Figure 7.14 — An excerpt of the report delivered automatically after generating a test 
case 


Our heuristic algorithm to optimize the objective function 


A. Process flow 


In conclusion, for a set of targets and weights on the coverage objectives and the time and 
cost constraints, the test engineer can generate one or more fest cases fulfilling these 
predefined objectives and constraints. Afterwards, the “Optimal” fest case 1s selected 
automatically. To do so, we compare the generated fest cases in pairs and we select the one 
which has the lowest value of the aggregated preference Fpjecrives Of the quality (coverage) 
indicators. If the two test cases have the same value of Fopjectives, We select the one which has 
the lowest value of the aggregated preference Foonstrainr Of the time and cost indicators. If the 
two test cases have the same value Of Fonstrain, We Select the utmost one that respects each 
individual constraints going from the higher to the lower weights. 


Moreover, a software functionality under test has often configuration signals (Cf. Chapter 5 — 
Section 3.4) which allow to parameterize the functionality (for instance, by activating or 
deactivating one feature of the functionality). A “Configuration” of a functionality consists to 
set all the configuration signals of the functionality to fixed values. Through our algorithm, 
two strategies are possible for managing the “Configuration” of a functionality. On the one 
hand, test engineers can generate one or more test cases for each specific “Configuration” of 
the functionality. The configuration signals are set to fixed values all over a test case. On the 
other hand, test engineers can generate one or more test cases Where each test case considers a 
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set of predefined “Configurations” of the functionality. In this case, the values of the 
configuration signals can change from one fest step to another within the same fest case. 


In Table 7.1, we describe the process flow of our optimization algorithm. The parameters of 
this algorithm are identified in Table 7.1 (Parameter i) and described in the next section. 
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Process flow 


Brief description 


Set a new “Configuration” for 
the functionality under test 


Fixed values on the 
configuration signals 


Set targets on the quality, time 
and cost indicators 


OfargetaNd Ciarget 


Setthe parameters of the 
optimization algorithm 


Parameters 


Choose a succession of two 
operations 


Operation 


Start generating a 
testcase 


aramete 
Optimize the coverage of 
the operation matrix? 


Yes 


Succession already 
covered? 


arameter 
N1 trials? 
Choose another 


succession of 
operations 


Test engineers can decide to test a functionality under 
different “Configurations” (Config1, Config? ….). 


We set a new “Configuration” for the functionality under test. 


Test engineers have to define the targets and weights for the 
coverage objectives and for the time and cost constraints 


Test engineers have to set the parameters of the 
optimization algorithm. 


We start generating a test step by choosing a succession of 
operations 


Did the test engineer decide to optimize the coverage of the 
‘operation matrix”? 


If NO, we keep the chosen operation 


If YES, we check if the chosen operation is already covered 
by another test step in the test case 


If NO, we keep the chosen operation 


f YES, we check if we already made N1 trials and all the 
chosen operations were already covered 


If NO, we choose a new succession operation 
If YES, we keep the last chosen operation 


To be continued … 
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Process flow 


Brief description 


Inter-operation time 


Simulate the requirements 
model and assess the 
expected results 


Test Step 
Assess the current values of 


the quality, time and cost 
indicators 


Ocurrent and Courrent 


Optimize the number of test 
steps? 


Fdecrease? 


N2 trials? 


Delete the test 
step and generate 
a new one 


We choose an inter-operation time within the time interval 


We set the chosen operation on the requirements model, we 
simulate the model during the inter-operation time and we 
assess the expected results on the output signals 


A new test step is generated 


We assess the current values of the coverage objectives 
and the execution time and cost constraints 


Did the test engineer decide to optimize the number of test 
steps in a test case”? 


If NO, we keep the generated test step 


f YES, we check if the aggregated preference F has 
decreased 


If YES, we keep the generated test step 


f NO, we check if we already generated successively N2 
test steps with no decrease of the aggregated preference F. 


f NO, we delete the last generated test step and we 
generate a new one 


If YES, we keep the last generated test step 
To be continued … 
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Process flow 


Brief description 


Add the generated test step to 
the test case 


Test Case 


Yes 
Fe objectives — we 


Fdecrease? 


Generate a 
new test step 


Parameter 
Last N3 Test Steps? 


Cumulate the coverage of 
different configurations? 


For a set of “Configurations”, we 
generate an “Optimal “Test Case 
fulfilling the predefined objectives 
by accumulating the coverage of 
the different “Configurations” 


For each “Configuration”, we 
generate an “Optimal “Test Case 
fulfilling the predefined objectives 


We add the test step to the current test case 
The current test case is updated with a new test step 


We check if the “set” coverage objectives have reached their 
targets (F objectives=0) 


f YES, we stop generating test steps for the current test 
case 


f NO, we check if the aggregated preference F has 
decreased 


If YES, we generate a new test step for the current test case 


f NO, we check if the aggregated preference F have not 
been decreased with the NS last test steps of the current test 
case 


If NO, we generate a new test step 


f YES, we stop generating test steps for the current test 
case 


Did the test engineer decide to cumulate the coverage of 
different “Configurations” in order to reach the targets on the 
coverage objectives”? 


If YVES, follow C 


If NO, follow D 
To be continued … 
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Process flow 


Brief description 


Other “Configurations” 
that should be tested? 


Choose the “Optimal” Test 
@ Case between the present test 
case and the previous one 


“Optimal “Test Case 


N4 generated Test 

Cases? 
Generate a 

new test case 


IS there other “Configurations” for the 
functionality that should be tested”? 


f YES, we change the “Configuration” 
and continue generating test steps for 
the same test case 


If NO, 
finalized 


the current test case is 


We choose the “Optimal” test case 
between the current test case and the 
previous one. 


The “Optimal” test case is chosen 


Do we generate NA test cases in order 
to choose the “Optimal” one? 


f NO, we generate a new test case for 
the set of “Configurations” 


f YES, we stop the algorithm and we 
deliver the “Optimal” test case for 
the set of “Configurations” 


Choose the “Optimal” Test 
Case between the presenttest 
case andthe previous one 


Optimal Test Case 


N5 generated Test 
Cases? 
Generate a 


new test case 
Yes 


Yes 


Other “Configurations” 
that should be tested”? 


The current test case is finalized 


We choose the “Optimal” test case 
between the current test case and the 
previous one. 


The “Optimal” test case is chosen 


Do we generate N£ test cases in order 
to choose the “Optimal” one? 


f NO, we generate a new test case 
with the same “Configuration” 


f YES, we check if there is other 
“Configurations” for the functionality 
that should be tested”? 


f YES, we change the “Configuration” 
and start generating a new set of test 
cases 


f NO, we stop the algorithm and we 
deliver an “Optimal” test case for 
each “Configuration” 


Table 7.1 —- Our heuristic algorithm to optimize the generation of test cases 
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B. Parameters 


In the Table 7.1, we identify 8 parameters that must be set by the test engineer before start 
generating fest cases: 


e  Parameter 1: Test engineer can decide to optimize the coverage of the “operation 
matrix”. To do so, Parameter 1 must be set to /. Otherwise, it is set to 0. 

e  Parameter 2: In order to optimize the coverage of the “operation matrix”, We check if 
the chosen succession of operations is already covered or not. When it is already covered, 
we propose to choose another succession of operations and so on. However, we have to 
avoid the non-stop loop in the algorithm. The Parameter 2 specifies the maximum 
number of unsatisfied trials (N]) before the algorithm exists the loop. 

e  Parameter 3: Test engineer can decide to optimize the number of fest steps in a test case. 
To do so, Parameter 1 must be set to /. Otherwise, it is set to 0. 

e  Parameter 4: …n order to optimize the number of test steps in a test case, we check if a 
generated test step decreases the aggregated preference F. In case of no decrease of F, 
we propose to delete the fesf step and generate a new one. However, we have to avoid the 
non-stop loop in the algorithm. The Parameter 4 specifies the maximum number of 
unsatisfied trials (N2) before the algorithm exists the loop. 

e  Parameter 5: In case of generating one or more test cases for each specific 
“Configuration” of the functionality, this parameter allows to stop generating fest steps 
for each fest case. In fact, we check if the aggregated preference F has not been 
improved on the last (N3) test steps of the current test case. If it is the case, we stop 
generating fest steps. It not, we continue generating rest steps. 

e  Parameter 6: Test engineer can generate one or more fest cases Where each fest case 
considers a set of predefined “Configurations” of the functionality. To do this, Parameter 
6 must be set to 7. When this parameter is set to 0, each generated rest case considers 
only one specific “Configuration”. 

e  Parameter 7 (used when Parameter 6 = 1): This parameter defines the number of test 
cases (N4) to be generated in order to identify the “Optimal” one. 

e  Parameter 8 (used when Parameter 6 = 0): This parameter defines the number of test 
cases (NS) to be generated in order to identify the “Optimal” one. 


VIL. Conclusion 


In automotive industry, the activity of designing manually test cases for software products 
becomes more and more laborious and time consuming. This activity accounts for more than 
50% of the total software project time and budget (Cf. Chapter 1 — Section 5.C.2). Despite the 
huge time and money spent in testing a software product and after each delivery to the 
carmaker, some bugs are detected by the carmaker. Since the late 90’s, the automation of the 
test case design process became a hot topic and automotive electronic suppliers are still 
looking for a relevant automation of this process. 


In this chapter, we developed our strategy to generate test cases automatically from our 
formal model to represent software functional requirements (Cf. Diagnosis 15). The selection 
of operations 1s performed based on a Monte Carlo simulation on an “operation matrix” (Cf. 
Diagnosis 13). AN the expected values on the output signals of the functionality are assessed 
through a simulation of the requirements model (Cf. Diagnosis 10 and 12). Moreover, test 
engineers could parameterize the generation of fest cases in order to take into account quality 
objectives but also time and cost constraints. Indeed, the decision to stop designing fest cases 
1s based on a formal measurement of the code and requirement coverage and the test time and 
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cost (Cf. Diagnosis 11). À heuristic algorithm 1s in charge of optimizing the generation of test 
cases While fulfilling quality objectives and constraints. 


In the following chapter, we suggest refining the operation space by focusing on critical 
operations Or succession of operations. To do this, we define driver behavior’s profile and 
propose to reuse bugs and test cases capitalized on similar projects in the past. 
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CHAPTER 8. REFINING THE OPERATION 
SPACE DESCRIPTION WITH THE DRIVER 
BEHAVIOR’S PROFILE, PAST BUGS AND TEST 
CASES 
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I. Introduction 


As automotive software products become more and more complex (Cf. Chapter 1), it 1s 
illusory to be able to check that the software product responds correctly to all possible 
operations. In other words, it is impossible to cover all the operation space of a software 
product (Cf. Chapter 2 — Section 6). In fact, each engineer has a different perception of the 
possible and critical operations (based on her/his experience). When designing test cases, test 
engineers aim to reach a code or requirement coverage objective. In fact, test engineers do not 
always select operations that simulate the real use of the software product under test. 
Moreover, they do not formally use capitalized bugs and test cases in order to improve the test 
design process on future developments. 


In this chapter, we point up how the operation space can be objectively refined by focusing 
on critical test scenarios. The complexity of testing exhaustively a software product is 
illustrated in Section 2. An overview on our operation space reducing techniques is proposed 
in Section 3. In Section 4, 5 and 6, we develop respectively each of these techniques: focusing 
on test scenarios regularly done by the end-user of the product, focusing on recurrent types of 
bugs through an analysis of the problems’ database and finally focusing on test engineers” 
experience feedback by reusing test cases capitalized on previous projects. 


IL. The impossibility of testing exhaustively a software product 


Testing exhaustively a software product is a NP-Complete problem from a computational 
viewpoint. In other words, it is very complex to test all the inputs, combinations of inputs and 
paths of a software. In computational complexity theory, the complexity class NP-Complete 
also known as NP-C or NPC, is a subset of the NP class ("Non-deterministic Polynomial 
time" class, (Karp 1972)). They are the most difficult problems in NP. To prove that an NP 
problem À 1s in fact an NP-Complete problem it is sufficient to show that an already known 
NP-Complete problem reduces to A. There are more than 3000 known NP-Complete 
problems. Most of the problems are listed in Garey and Johnson's seminal book (Garey 1979). 
In (Seroussi 1988), Seroussi and Bshouty prove that the design of an optimal exhaustive test 
case for an arbitrary logic circuit is an NP-Complete problem. In fact, they demonstrate that 
finding the minimal rest case (and its size) which covers the logic circuit is an NP-Complete 
problem. In order to do this, they first show that the problem can be solved by a 
nondeterministic algorithm in polynomial time. Then, they use the standard technique of 
reduction to prove that the problem is NP-Complete: for a given problem P (the Graph 
Coloring problem” *) known to be NP-Complete, they show that if our problem is solvable in 
deterministic polynomial time, then so is P. In (Cheng 2003, Hessel 2007), the authors prove 
that finding optimal test cases for a software product is an NP-Complete problem. Indeed, 
they reduce the problem of generating test cases to the set-covering problem (an NP-Complete 
problem). 


For our research project, we propose to generate automatically rest cases for a software 
product. Our approach is based on modeling the software functional requirements and 
generating fest cases from this model. To guide the design of fest cases, code and/or 
requirement coverage criteria are used. À coverage criterion can be seen as a set of items 
(relations between inputs and outputs) in the source code or requirements model to be 
covered. Therefore, our test generation problem can be formulated as a Reachability 


# Graph coloring problem, http:/en.wikipedia.org/wiki/Graph_coloring (Consulted on November 2008) 
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problem” (an NP-Complete problem) which consists to explore the operation space only if it 
might increase the total coverage. In fact, a test case is a set of ((TSZ,Cov1), (TS2, Cov2), 
(TS3, Cov3), .… (TSn, Covn)), where TSi is a test step and Covi is the coverage contribution 
performed by TSi. Ideally, this set should be reduced so that the total coverage ZCovi is not 
changed, and the length of the rest case, e.g. Z]TSi] is minimized. However and as it was 
shown above, designing a subset of fest steps With this property is an NP-Complete problem 
(the Reachability problem). 


A present, all known algorithms for NP-Complete problems require time that is 
superpolynomial (for instance, exponential) in the inputs size, and it is unknown whether 
there are any faster algorithms. The following techniques can be applied to solve 
computational problems in general and they often give rise to substantially faster algorithms: 


e  Randomization: Use randomness to get a faster average running time, and allow the 
algorithm to fail with some small probability. 

e  Heuristic: An algorithm that works "reasonably well" on many cases, but for which 
there is no proof that it is both always fast and always produces a good result. In 
Chapter 7 — Section 6, we develop the heuristic algorithm that we use in order to 
explore the operation space of a software product and monitor the generation of test 
cases. 


In the next sections of this chapter, we develop how to reduce the operation space of a 
software product by highlighting and eliminating some operations or succession of 
operations. Our purpose 1s to explore the operation space of a software product efficiently. 


III. Reduce the operation space 


As said in the previous section, selecting operations from the whole operation space in order 
to reach a coverage objective is a NP-Complete problem. For that reason, it can be useful to 
reduce the operation space by: 


A. Focusing on recurrent operations done by the end-user of the product 


We analyzed in 2006 a set of software bugs (the number of these bugs is confidential) 
detected on different types of products by carmakers and end-users (drivers) and not detected 
by Johnson Controls. The conclusion which was validated by Johnson Controls software 
experts 1s that some of these bugs (more than 50%) can only be detected via successions of 
operations regularly done by the end-user of the product. Therefore, we propose to generate 
test cases that simulate the behavior of the end-user of the product. To do so, test cases must 
be generated from “operation matrices” Where illogic (from end-users’ viewpoint) 
successions of operations are eliminated (for instance, set the vehicle speed at 100 km/h then 
open the trunk) and regular successions of operations are favored (for instance, close the 
driver door and set the ignition). Our process to define a driver behavior’s profile is 
developed in Section 4. 


B. Focusing on specifics operations with high probability to detect bugs 


We performed in 2007 a study on 70 software bugs detected on the same functionality 
developed in Johnson Controls for 5 different projects respectively in 1997 (27 bugs), 2001 (4 


# Reachability problem, http://en.wikipedia.org/wiki/Reachability (Consulted on November 2008) 
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bugs), 2003 (4 bugs), 2003 (13 bugs) and 2007 (22 bugs) (Cf. Chapter 2 — Section 7.C). The 
studied functionality has 7 features, so we classified these bugs by project and by feature (Cf. 
Figure 2.22). We also classified these bugs by project and by type of problem (Cf. Figure 
2.23). In fact, we only consider 4 types of problem (Beizer 1990, Chillarege 1992, Grady 
1992, IEEE Std. 1044-1993): code implementation, control flow, data and processing. A full 
typology of software bugs is described in Section 5.B. The conclusions which were validated 
by Johnson Controls software experts are 1) engineers have the tendency to make errors in 
implementing the same features of a functionality (Feature 1, 3 and 7) and 2) these errors are 
related to the same types of problem (Control flow and processing). As a consequence and 
when testing a functionality, we propose to reuse related stored bugs in order to generate test 
cases Which verify the nonexistence of recurrent bugs. To do so, test cases must be generated 
from “operation matrices” Where the successions of operations that reveal the recurrent bugs 
are favored. Moreover, classifying the recurrent bugs of a functionality (by feature and/or by 
type of problem) could help the test engineers to better focus the generation of fest cases on 
critical features or on specific types of problem. More details on reusing stored bugs are 
provided in Section 5. 


C. Focusing on specifics operations recurrently done by the test engineers 
on previous projects 


Using capitalized test cases seems to be beneficial in automotive context since more than 50% 
of functionalities performed by software products are common to any series of cars (Johnson 
Controls source). Moreover, test cases management and reuse are considered as one of the 
main characteristics of a mature software organization. Therefore, when testing a functionality 
that we already implemented in the past on another project, it is judicious to reuse existing fest 
cases. To do so, we propose to analyze test cases developed in the past for the same 
functionality and design “operation matrices” where the operation space is reduced by 
focusing on the test scenarios based on our returns of experience. Test cases generated from 
these “operation matrices” contain similar successions of operations as in the one designed 
manually or generated automatically in the past. More details on reusing capitalized test cases 
are provided in Section 6. 


IV. Four types of constraints for the definition of a driver behavior’s 
profile 


We define four types of constraints that test engineers can affect to each input signal of a 
requirements model in order to, when generating test cases automatically, eliminate or favor 
specific successive operations. Each input signal can have one or more constraints. These 
constraints aim to reduce the number of possible combinations on input signals and to more 
thoroughly pinpoint which ones are frequently set once the product is launched on the market. 
These four constraints are: logical constraint, conditional constraint, succession constraint 
and timing constraint. 


A. Logical constraint 


This constraint forbids that an input signal switches between inadequate values from a use 
point of view. In order to illustrate this constraint, let us consider the input signal 77, which 
has a domain D(11)={0,1,2,3}. We classify input signals into two types: 


Acyclic (Cf. Figure 8.1): Input signal I] is acyclic if, at any moment, all the operations (11=0 
or /1=1 or 11=2 or 11=3) on the signal are possible. A practical example of an acyclic signal 
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is the rain intensity signal measured via a sensor. When it is raining so hard (/7=3), it can 
stop raining (//=0) at any moment without a decrease of the raining intensity (/7=3 — 11=2 
— 11=1 — 11=0). 


Figure 8.1 — Acyclic signal 


Cyclic (Cf. Figure 8.2): Input signal I is cyclic if the future operation (11=0 or 11=1 or 11=2 
or /1=3) on the signal depends on the one did in the past. A practical example of a cycle 
signal 1s the wipers’ switch signal. When wiping at a high speed (//=3), user cannot 
immediately switch off the wipers (/7=0) via the switcher. In fact, she/he must progressively 
slow down the wiper speed until the complete stop (/1=3 — 11=2 — 11=1 — 11=0). 


Figure 8.2 — Cyclic signal 
B. Conditional constraint 


This constraint characterizes a specific user behavior between two or more correlated input 
signals. In other words, when one or more inputs fulfill specific conditions, the domain of 
other inputs is adapted (shrinked) automatically. For instance (Cf. Figure 8.3), the vehicle 
speed cannot be more than 0 (770), only if the vehicle is running (/2=]) and the vehicle can 
be running (/2=]1) only if the car engine is switched on (13=1). 


11 cannot be more than 0 Only if 12 is equal to 1 


2 
= 
12 cannot be equal to 1 | Only if 13 is equal to 1 


DR PRE | M Rs | 
FR EE 


Figure 8.3 - Conditional constraint 


12 
=1 
12 
=1 


C. Succession constraint 


In practical use of an electronic product, two or more operations have a high probability to 
succeed (sometimes, must intuitively succeed). Through this type of constraint, we favor such 
successive operations. For example (Cf. Figure 8.4), when drivers close the driver door 
(11=1), they often (with a probability of 0,75) switch on the car engine (/3=1). 
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Probability only 
Figure 8.4 — Succession constraint 


D. Timing constraint 


Major Johnson Controls software experts approve that time interval between operations plays 
a major role in bugs’ detection. On the one hand, two specific operations can be performed 
with a specific time interval (Cf. Figure 8.5). For instance, in case of a taxi driver, the driver 
door is closed (/7=1) and the car engine is switched on (/3=1) within a small time interval 
([S0ms*, 100ms] according to experts). 


Inter-operation time interval only 
Figure 8.5 — A specific time interval between two operations 


On the other hand, a specific operation can be performed during a specific time (Cf. Figure 
8.6). For instance, the ignition 1s switched off (13=0) for more than 5 seconds (according to 
experts) in order to reset a functionality. 


Inter-operation time interval only 
Figure 8.6 — An operation set during a specific time 


V. Reuse of bugs detected on previous projects 


Each bug stored in the bug’s database has a set of 111 attributes “theoretically” filled by the 
engineer while resolving the bug. In Chapter 2 — Section 7.B, we estimate that 75% of these 


36 ee 
ms: millisecond 
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attributes are filled; the remaining 25% are systematically unfilled. On the 75% filled 
attributes, 25% of these attributes are free fields. Moreover, we deduce that the problems’ 
database in Johnson Controls is mainly used to manage the bugs and keep their traceability. 
Unfortunately, none of the 111 attributes is useful to identify critical succession of operations 
or recurrent types of problem for a specific functionality. In this section, we propose two 
strategies in order to reuse bugs detected on previous projects. The first strategy consists of 
defining a specific format to capitalize the initial conditions and the successive operations 
which lead to detect a bug. The second strategy aims to define a detailed typology of software 
problems. 


A. A specific format to capitalize the successive operations leading to a bug 


Now at Johnson Controls, engineers describe how the bug was detected by filling a free field 
in the problems’ database named “Problem description” (Cf. Figure 2.19). Indeed, apart the 
requirement of using the English language, no other requirements or recommendations for 
flling this field are given to database users. In fact, each engineer has to describe the way the 
bug was detected by giving as much detail as possible. In Figure 8.7, we propose a new 
specific format to describe the successive operations leading to a bug. 


Problem description 


The initial state of the functionality under test. Here, practitioner 
must list all the initial values on the functionality input signals 


The first operation. Here, practitioner must put nothing or an input 
signal set to a specific value 
The waiting time before performing the second operation. Here, 
practitioner must put a time in millisecond 
The expected values on the functionality output signals. Here, 
Expected values on output signals practitioner must list all the expected values on the functionality 
output signals 
The observed values on the functionality output signals. Here, 
practitioner must list all the observed values on the functionality 


output signals 


Inter-operation time (ms) 


operation 


nter-operation time (ms) 
xpected values on output signals 
Observed values on output signals 


Figure 8.7 — À predefined format to fill in the “Problem description” attribute of a bug 


Let us consider a practical example of a functionality with 3 input signals (11, Domain = {0, 
1}; 22, Domain = {1, 2, 3}; 13, Domain = {0, 1 }) and two output signals (O1, Domain = {0, 
1}; O2, Domain = {0, 1}). When testing this functionality on a project in 2005, a test engineer 
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has detected a bug and added this bug to the bug’s database. In Figure 8.8, we illustrate how 
the “Problem description” attribute of this bug was filled in. In Step 6, the observed values on 
output signals are different from the expected values (this is the symptom of the bug). For 
each bug in the database, the functionality where the bug was detected 1s specified. Therefore, 
when testing the same functionality on a project in 2007 (2 years after), a test engineer could 
select from the problems’ database all the bugs detected on this functionality on previous 
projects. Based on the predefined format of the “Problem description” attribute, each bug can 
be translated automatically into an “operation matrix” (Cf. Figure 8.8). À glossary of the 
input signals names on the previous and current projects 1s necessary. In fact, from one 
project to another, the name of an input signal can change even if the use of the signal stills 
the same. The test cases generated from this “operation matrix” (Cf. Figure 8.8) allow to 
check if the bug that we detected 1in the past 1s present or not in our new development of the 
functionality. 
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À capitalized bug 


Problem description 


M=1 
Initial values on input signals 12=1 
13=0 
Step 1 
st operation 11=0 
Inter-operation time (ms) 50 
Expected values on outputs O1=0; O2-0 
Observed values on outputs O1=0; O2-0 


Step 2 


2nd operation 


Inter-operation time (ms) 


200 


Expected values on outputs O1=0; O2-0 
Observed values on outputs O1=0; O2-0 
Step 3 

8rd operation 12=2 
Inter-operation time (ms) 100 
Expected values on outputs O1=1; O2-0 
Observed values on outputs O1=1; O2=0 
Step 4 
4th operation 13=1 
Inter-operation time (ms) 800 
Expected values on outputs O1=1; O2=1 
Observed values on outputs O1=1; O2=1 
Step 5 
5th operation 11=0 
Inter-operation time (ms) 200 
Expected values on outputs O1=0; O2=1 
Observed values on outputs O1=0; O2=1 
Step 6 
6th operation 
Inter-operation time (ms) 300 
Expected values on outputs Oi=1; O2=1 
Observed values on outputs O1=1; O2=1 


Step 7 


7th operation 


13=0 


Inter-operation time (ms) 


150 


Expected values on outputs 


Observed values on outputs 


1-1 has succeeded to 
11=0 with two different 
inter-operation times 


The generated 


“Operation matrix” 
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(50ms and 200ms) 1 12=2 has ir il 18=0 has I 
D t-------- a 1--! | succeededone 1 | Succeededone 1 
À \ 1 1 timeto 11=1 (50%) ! 1 timeto 11=1 (50%) ! 
(l \ CCE = rs CPE J 
Ï / 
I M LE RP [ 
| U 1 1 2 23 0 1 
° 0 [woza| © op 0 NT à 
0,5 0,5 
d 5 0 5200200! © 300300] © 
0 0 0 0 0 0 0 Automatic 
ï generation of 
0 L J È 0 9 |1100.104| | “Test Cases’ 
0 0 0 0 0 0 0 
o| o 0 0 0 0 0 0 
: {] 
: EE © 0 0 0 0 0 


R. AWEDIKIAN 


The generated 
“Test Cases” 


Test Case 1 


Test Ste : 
No P Test Actions | Expected Results 
1 1=0 To be filled by simulatin 
Wait 200 ms the requirements model 
2 1=1 To be filled by simulatin, 
Wait 200 ms the requirements model 
12 = 2 To be filled by simulatin 
3 ; the requirements model 
Wait 100 ms 
13 = 1 


Wait800ms Test Case n 


Figure 8.8 — Process of reusing bugs capitalized in the problems” database 
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Test Ste] à 
No p Test Actions | Expected Results 
1 1=0 To be filled by simulating 
Wait 100 ms the requirements model 
2 1=1 To be filled by simulating 
Wait 300 ms the requirements model 
13 = 0 To be filled by simulatin 
3 | the requirements model 
Wait 800 ms 
4 1=0 To be filled by simulating 
Wait 50 ms the requirements model 
5 1=1 To be filled by simulating 
Wait 200 ms the requirements model 
6 2=2 To be filled by simulating 
Wait 100 ms the requirements mode 
7 13=1 To be filled by simulatin 
Wait 800 ms the requirements model 
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In conclusion, for each functionality, test engineers can generate a set of test cases from the 
bugs detected on the same functionality on previous projects. Once executing these rest cases 
on the new software of the functionality, test engineers can, at least, guarantee that the new 
development is free of the bugs already made in the past. 


B. A new typology of software problems 


A second way to reuse bugs stored in the problems’ database is to analyze these bugs and 
identify the recurrent type of problems when implementing a software product. The “Problem 
type” captures the nature of the fix. It addresses the question: “What did the engineer correct 
in order to resolve the bug?” With this definition of “Problem type”, the classification is 
easier for the engineer, since she/he can almost decide objectively which attribute value to 
assign. Presently, Johnson Controls doesn’t have a typology of software problems (Cf. 
Chapter 2 — Section 7.B). 


In 2007, we participated to a work group on the definition of a new bug classification model 
within Johnson Controls. The aim of this new model is to be able to identify process 
improvement actions for the development and Verification and Validation (V&V) processes. 
In other words, the new bug classification model must answer the question of “which types of 
bugs are injected and detected in which process phase?” The classification model currently 
used in Johnson Controls (Cf. Chapter 2 — Section 7.B) does not allow to answer this question 
since there is no typology of software problems. The new bug classification model that we 
propose is summarized in Figure 8.9. It is mainly inspired by the literature on the subject (Cf. 
Chapter 4 — Section 4.C.2). The new model is based on three attributes: 


e Detection phase: the phase of the process where the bug was detected. 

° Injection phase: the phase of the process where the bug was injected. 

e  Problem/Correction type: this attribute answers the question of “what did you fix in 
order to correct the bug?” 


For each of these attributes, a set of predefined values was defined in accordance with the 
Johnson Controls software process (Cf. Chapter 2): 


e Detection phase: Specification review, Design review, Code review, Component test, 
Integration test, Manufacturing test, Validation test, System test, Test review, 
Customer test. 

e Injection phase: System Specification, Specification, System Design, Design, 
Implementation, Integration, Manufacturing, Testing phases (Component test, 
Integration test, Validation test, System test, Manufacturing test, Customer test). 

e  Problem/Correction type: À two-level typology of software problems is defined. The 
values of the first-level typology are: Specification update, Design update, 
Implementation update, Integration update, Manufacturing update, Test case update, 
Update none. The second-level typology 1s detailed in Appendix E. 
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Detection | |[Specificati[| Design Code Component Integration|| Manufact|| Validati || System Test || Customer 
phase onreview || review review test test uringtest || ontest test review test 


Injection System Specificat System Design Impleme ntegration Manufac Testing 
phase Spec. lon Design ntation turing phases 


DRE ESS 


Problem Specificati| Design  |Implementalntegration |Manufactur| Test Case | Update 
Pie onupdate| update |tionupdate| update |ingupdate|| update none 


Problem 


Figure 8.9 — Our new bug classification model 


On the one hand, as in HP classification model (Chillarege 1992); the attributes /njection 
phase and Problem/Correction type are linked. Indeed, the choice of a value for the attribute 
Injection phase defines the possible values for the attribute Problem/Correction type. Xn 
practice, when an engineer is editing a bug in the problems’ database, once she/he defines the 
Injection phase of the bug, the list of proposed Problem/Correction type must be adapted 
dynamically to the chosen Injection phase. For example, if the bug was injected in the 
Specification phase, the list of Problem/Correction type must be (Cf. Appendix E): 
requirements incorrect, requirements logic, requirements completeness .. If the bug was 
injected in the /mplementation phase, the list of Problem/Correction type must be (Cf. 
Appendix E): data definition, structure, declaration, data access and handling, control flow and 
sequencing … 


On the other hand, at the end of each design phase, one or more V&V phases have the 
responsibility to detect all the type of problems injected in this phase (Cf. Table 8.1). The 
validation and system test are the last V&V phases before the software product delivery to the 
carmaker. These phases have to check the compliance of the developed software with the 
carmaker requirements. They have the responsibility to detect all the type of problems 
injected during the design and development phases and not detected by the corresponding 
V&V technique. 


Must be detected in 
System specification Specification review 
Validation and System test 


Validation and System test 
Code review 
Implementation Component test 
Validation and System test 
Validation and System test 


Manufacturing test 


Manufacturing Validation and System test 


Table 8.1 - Theoretical bugs’ injection and detection phases 


However, some V&V activities (Detection phases) are not enough reliable to detect all related 
types of problems. The new bug classification model enables to pinpoint such lacks in the 
software process. In fact, managers can identify the density and type of software problems 
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injected in and detected by each phases of the software development life cycle. Therefore, 
improvement actions on the design phases (develop new design or development rules ...) and 
on the V&V phases (develop new code review rules, a new testing strategy...) can be 
performed. The first-level typology of software problems 1s not enough detailed and the 
improvement actions that can be raised from will not be enough efficient. For instance, after 
analyzing a set of bugs related to Specification Update problems, one engineer can note that 
the total number of these bugs is injected in Specification System and Specification phases and 
few of them are detected in Specification Review phase. As a conclusion, the Specification 
Review process has to be improved. However, it is a vague action. Engineers need to know 
what they have to improve in the review process. Are the requirements unclear? Incorrect? Do 
the requirements change often? ..…. We work in collaboration with software experts in order to 
define the second-level software problem typology. Based on the literature review (Beizer 
1990, Chillarege 1992, Grady 1992, IEEE Std. 1044-1993) and taking into account the 
automotive and industrial context, we propose in Appendix E a detailed typology of problems. 


Through our research project, a new approach to generate fest cases automatically for a 
functionality 1s proposed. The generated fest cases have the responsibility to detect all the 
bugs injected during the /mplementation phase of the source code. Analyzing the bugs 
injected during the /mplementation phase of the same functionality on previous projects 
allows test engineers to better parameterize the generation of fest cases. Instructions to 
generate fest cases able to detect one specific type of software implementation problems (Cf. 
Appendix E) are listed in Table 8.2. 


Types of software Can this type of Instructions when generating test 
implementation problems |problem be detected by|cases in order to detect this type of 
(Cf. Appendix E) test cases? problem 


Cover at 100% the input signals domain, 
output signals domain, intermediate signals 
Data definition, structure, domain 
declaration Cover at 100% the inputs boundaries 
domain, outputs boundaries domain, 
intermediates boundaries domain 


Cover at 100% the input signals domain, 
Data access and handling output signals domain, intermediate signals 
domain 


Cover at 100% the code conditions and 
decisions 
Cover at 100% the FSM transitions and 
conditions 
Cover at 100% the input signals domain, 
output signals domain, intermediate signals 
domain 
Cover at 100% the code procedures 
Cover at 100% the DT conditions 
Cover at 100% the FSM states 
oding and typographical YE Cover at 100% the code statements 
NO, code review or other 

V&V techniques 
NO, code review or other 

V&V techniques 


Table 8.2 — Instructions to generate test cases able to detect one specific type of software 
implementation problems 


Control flow and sequencing 


Processing 


Standards violation 


Documentation 
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VI. Reuse of existing test cases from previous projects 


Test cases on previous projects are versioned by software functionality and stored in a 
database. But unfortunately, these rest cases are not always reused from one project to 
another. Two main reasons are identified (Cf. Chapter 2 — Section 8): 1) the use of different 
formats when designing manually test cases. Sometimes, test engineers write the rest cases 
immediately in a computer language (C language, test script) understandable by the test 
execution platform. Others use the fest case format presented in Definition 2.11. 2) the lack of 
an automated process to reuse these rest cases. However, one initiative was launched two 
years ago and had the purpose to create manually “Standard Test Cases” for software 
validation (Cf. Chapter 2 — Section 8). This is a conventional RETEX (RETurn of EXperience) 
strategy and the main difficulty of such an approach is to keep these rest cases updated. Two 
years after, 1t 1s not the case. 


Through our research project, we adopt the fest case format presented in Definition 2.11 as the 
standard format to represent a test case. Our proposal to reuse existing fest cases on previous 
projects is based on this assumption. When testing a functionality, test engineers could select 
from previous projects all the rest cases related to the functionality under test. A glossary of 
the input signals names on the previous and current projects is necessary. Then, for each test 
case (independently from the length of the fesf case), an “operation matrix” 1s generated 
automatically. This “operation matrix” has high probability on the successive operations 
regularly done in the test case. It also contains the set of inter-operation time used between 
each couple of operations. Consequently, when generating fest cases from these “operation 
matrices”, We reduce the operation space by focusing on the test scenarios based on our 
returns of experience. 


Let us consider a practical example of a functionality with 3 input signals (11, Domain = {0, 
1}; 22, Domain = {1, 2, 3}; 13, Domain = {0, 1 }) and 2 output signals (OI, Domain = {0, 1}; 
O2, Domain = {0, 1}). This functionality was already developed on a previous project in 2005 
and one fest case has been designed. Therefore, when testing the same functionality on a 
project in 2007 (2 years after), a test engineer selects from the database the test case already 
designed in the past (2005) for this functionality. Each fest case can be translated 
automatically into an “operation matrix” (Cf. Figure 8.10). The test cases generated from this 
“operation matrix” (Cf. Figure 8.10) focus on specific test scenarios that test engineers have 
judged critical to perform in the past. 
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À Capitalized test case The generated The generated 
“Operation matrix” “Test Cases” 


Test | Test Actions | Expected Results 
Step No JL 
Wait 250 ms O2=1 
: [5 Î O1=0 Test Case 1 
Wait 200 ms | O2=1 
: B=T Ot=1 . Test Actions | Expected Results 
Wait 300 ms O2=1 1 1=0 To be filled by simulatin 
: 12 = 1 OC Wait 100 ms | ‘e"equiremients modé 
Wait250 ms O2=1 12=1 has : 12=2 has 13=3 has 2 18=0 To be filed by simulaing 
5 |E=$ O1=0 l succeededone || succeededone 1 | succeededone 1 Wait 200 ms | (e"eduirenients modé 
Wait 900 ms | 02-0 Re Pen A | nn ere 
1 =0 O1=0 s + —+ ait 350 ms 
6 LE 12 Brie 
. : 2=2 
Ba T ot : 5 - , | 1: a : 05 ; 5 u Wait200ms Test Case n 
7 : = 9 Mu 9 ° y 0 00100 200.200 | 
Wait 400 ms O2=1 0 . 0,33 0,33 0 L 0,33 5 Test Step | 
Poe — ” Dis] [100,100] | [250,250] [100,100] | No Test Actions | Expected Results 
8 : 0 0 0 0 1 0 0 Automatic 
fateqo me = 1 GE HS0,2501 55 generation of 6 n - 0 Tobe filed by simulating 
9 = 01=0 2]100700| _ ° É 9 à 9 |1200,200 || ‘Test Cases’ Wait 200 ms | "2794780878 00e 
Wait 100 ms O2=1 0,33 0,66 7 3=1 - —— 
. = — O1= 0 SRE 00,500 S 9 o L û Wait 300 ms ane mao 
; 110 0 0 0 0 0 _ ———. 
Muaiisome) 2-0 RE CE 
11 de =3 Il O1=0 0, “ lrsoosoo/ 1350350 / 1400409! © À Wait 200 ms 
Wait 500 ms O2=1 Æ, A 1B=1 To be filed by simulating 
12 NM = 1 O1=1 ï és Pneus s : Wait 400 ms | ‘€ "equiremients modèl 
Wait 100 ms O2=1 =1hassucceededto ! * 12= 3 ; imulati 
PU LEE Gi=0 RS * Wait 900 ms | éreaurements model 
: à (l 
[Wait 350 ms | O2=1 (200ms and 500ms) 1 1=0 To be filed by simulating 
12 = 2 O1=0 TPS SE RS Re RE ER ! Wait 400 ms | #e’equiremients mode 
14 : 
Wait 700 ms O2=0 18 = 0 To be filed by simulating 
 =0 O1=0 Wait 100 ms the requirements model 
15 
Wait 100 ms O2=1 
13 = 1= 
16 3 . O1=0 
Wait 200 ms O2=0 
17 I =0 O1=0 
Wait 200 ms O2=1 


Figure 8.10 — Process of reusing test cases capitalized on previous projects 
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VIL. Conclusion 


Only exhaustive testing can show that a software product is free from bugs. However, 
exhaustive testing of a software product is not practical because variable input values and 
variable sequencing of inputs result in too many possible combinations to test. So it would be 
useful to concentrate the test on the areas associated with the greatest risks and priorities. 
However, we have to identify and analyze these risks and priorities. 


In this chapter, we developed three strategies able to reduce the operation space of a software 
product. Our main purpose was to focus on test scenarios with a high probability to detect 
software bugs. Firstly, we specified four types of constraints that test engineers can set on the 
input signals of the functionality under test in order to favor or avoid specific successions of 
operations. Secondly, we developed a new “Problem description” format to capitalize the 
initial conditions and the successive operations that lead to a bug. Based on this new format, 
tester engineers can generate automatically one or more test cases from each capitalized bug. 
We also developed a detailed software problem typology that helps test engineers to identify 
recurrent types of problems and better address the generation of test cases. Finally, we set up 
an automatic process to reuse fest cases from one project to another. 


In the latest four chapters (Chapter 5, 6, 7 and 8), we specified our approach to improve the 
global performance of the Johnson Controls V&V activities. In the following two chapters 
(Chapter 9 and 70), we respectively implement our approach in a computer platform 
(prototype) and validate it through two industrial case studies. 
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CHAPTER 9.  PROTOTYPE IMPLEMENTATION 
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I. Introduction 


After specifying our new approach to generate efficient test cases automatically (Cf. Chapter 
5, 6, 7 and 8), we focus in this chapter on the practical use of this approach within an 
industrial context. We develop a prototype implementing our models, concepts and theories. 
A “functional” view of our approach is illustrated in Section 2. À “process-role-tool? view of 
our approach is proposed in Section 3. The processes are mainly defined in Chapter 5, 6, 7 
and 8. Some specific skills which are mandatory when using our approach are detailed. We 
also describe the three computer tools that we developed in order to automate the generation 
of test cases. The main one is the Test Case Generation tool which is a PC application. The 
main functionalities of this tool are developed in details in Section 4. 


IL. A “functional” view of our approach 


In Chapter 2 — Section 6, we describe how Johnson Controls test engineers currently design 
test cases for software products. They proceed to a manual design of test cases. The 
performance of the design is mainly based on their experience. In Chapters 5, 6, 7 and 8, we 
develop our new approach to design fest cases automatically. À functional view of our 
approach is presented in Figure 9.1. It is based on eight activities. These activities are: 


1. Design a simulation model of the software functional requirements of the functionality 
under test (Cf. Chapter 5). 

2. Verify and validate the requirements model (Cf. Chapter 6). 

3. Define some behavioral characteristics of a car driver when using the functionality 
under test (Cf. Chapter 8). 

4. Perform a statistical analysis on bugs detected in the past on the functionality under 
test (Cf. Chapter 8). 

5. Perform a statistical analysis on test cases developed (in the past) on the functionality 
under test (Cf. Chapter 8). 

6. Highlight the relevant, critical and mandatory operations and succession of operations 
to be chosen from the operation space of the functionality under test (Cf. Chapter 8). 

7. Automate the design of test cases from the requirements model (Cf. Chapter 7). 

8. Monitor the design of fest cases by quality objectives and time and cost constraints 
(Cf. Chapter 7). 
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Figure 9.1 — A “functional” view of our approach 
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III. A ‘“process-role-tool” view of our approach 


Our approach presents a much different workflow for designing fest cases than the present 
one. The new workflow is presented in Figure 9.2. It is composed from seven processes 
which are manual, semi-automatic or automatic and managed by different individuals 
(experts, modelers and test engineers). 
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Figure 9.2 — A “process-role-tool”? view of our approach 
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A. 


Processes 


Our approach is composed of seven processes: 


1. 


2 


à) 


Modeling process (manual): models the software functional requirements using our 
formal specification language. 

Driver profile definition process (manual): defines the driver behavior when using the 
functionality under test. 

Bugs reuse process (semi-automatic): establishes a framework in order to reuse the 
bugs capitalized in the problems’ database and related to the functionality under test. 
Test cases reuse process (semi-automatic): establishes a framework in order to reuse 
the test cases developed on previous projects and related to the functionality under 
test. 

Model verification and validation process (automatic): verifies and validates the 
requirements model consistency and compliance with the carmaker requirements. 
Model simulation process (automatic): simulates the requirements model 

Test Case generation process (automatic): monitors the generation of fest cases by 
quality objectives and time and cost constraints 


These seven processes have been developed in details in Chapters 5, 6, 7 and &. 


B. 


Roles 


“Roles” can be allocated to one or more people in a software project provided one has the 
time and the required skills. Three types of roles have been identified: 


Modeler: the main tasks of a modeler are to analyze the software functional 
requirements, design the requirements model, verify and validate the model and finally 
simulate it. À modeler has to be familiar with the behavior of the car’s software 
functionalities and a master of the formal specification language. She/he also needs 
good communication skills. Indeed, she/he has to interact with the carmakers in order 
to eliminate ambiguities and inconsistencies from the requirements. Finally, analysis 
skills are necessary for the Verification and Validation (V&V) of the requirements 
model. 

Expert: the main tasks of an expert or a group of experts are to define a driver profile 
for the functionality under test, to identify related bugs and fest cases capitalized on 
previous projects and to extract from these bugs and fest cases relevant lessons 
learned. An expert has to be a master of automotive electronics. She/He needs to have 
a global view of all the projects and software practices within the company. 

Test engineer: the main tasks of a test engineer are to parameter1ze the test generation 
algorithm and to set the quality objectives and the time and cost constraints. The 
generation of fest cases is automatic. However, test engineer has to execute the 
generated fest cases on the software product under test and analyze the results. A test 
engineer has to be familiar with the behavior of the car’s software functionalities and 
the formal specification language. Knowledge about optimization is necessary to 
better parameterize the optimization algorithm. In addition, she/he has to be a master 
in requirements and code coverage in order to set relevant coverage objectives. 
Finally, analysis skills are mandatory for the analysis of the test results and reports. 
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C. Tools 


In this section, we develop the three computer tools supporting the semi-automatic and 
automatic processes of our approach: Bugs Reuse tool, Test Cases Reuse tool and Test Case 
Generation tool. 


1. Bugs Reuse tool 


In order to reuse the bugs capitalized in the problems’ database, experts have to identify the 
relevant bugs related to the functionality under test. In Chapter 8 — Section 5.B, we define a 
new format to fill in the “Problem description” attribute of a bug. Based on this format, we 
develop an Excel Macro able to analyze the “Problem description” of a bug and to generate 
the corresponding “operation matrix” (Cf. Figure 9.3). This matrix is used to generate test 
cases able to detect a similar bug on future development. When analyzing a bug and 
generating an ‘operation matrix”, the Excel Macro uses à glossary of input signals names on 
the previous and current projects. The Macro has been developed in Visual Basic language. 


Bug 


Problem description 
n 


Initial values on input signals = 
13= 


Step1 
[Tstoperation n=0 
nter-operation time (ms) 50 
[Expected values on outputs | O1=0; 022 
[Observed values onjoutputs 59] 01-0: 02 


Rss Operation matrix 
= — Excel Macro 1 à 


(Observed values on outputs 01=0; 02=0 nm 12 13 


Step 1 B 0 


Bra operation 1-2 6 5 


1 
Inter-operation time (ms) +00 [50,200] 
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0 
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Q 
Q 
Q 
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Q 
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Step [800,800] L 


(IR operation = 
Înter-operation time (ms) 300 
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values on outputs | 01=1; 02=1 
Step7 
(th operation 18=0 
Inter-operation time (ms 150 
[Expected values on outputs 
[Obs values on outputs | 


Figure 9.3 —- Bugs Reuse tool 


A detailed description of the process of analyzing the bug and generating the “operation 
matrix” 1s given in Chapter 8 — Section 5.A. 


2. Test Cases Reuse tool 


In order to reuse the test cases from one project (in the past) to another (in the present or 
future), experts have to identify the test cases related to the functionality under test. In 
Chapter 8 — Section 6, we adopt the test case format defined in Definition 2.11. Based on this 
format, we develop an Excel Macro able to analyze a test case and generate the corresponding 
“operation matrix” (Cf. Figure 9.4). In this matrix, the operation space is reduced by 
focusing on the test scenarios based on the returns of experience. Test cases generated from 
this “operation matrix” contain similar successions of operations as in the one designed 
manually or generated automatically in the past. À glossary of the input signals names on the 
previous and current projects is also necessary. The Macro has been developed in Visual 
Basic language. 
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Figure 9.4 — Test Cases Reuse tool 
A detailed definition of the process of analyzing the fest case and generating the “operation 
matrix” is given in Chapter 8 — Section 6. 


3. Test Case Generation tool 


Through our research project, we were asked by the automotive electronic supplier Johnson 
Controls to automate the design of rest cases for software products (Cf. Chapter 1 — Section 
6). Therefore, we develop a computer tool, the Test Case Generation tool, able to computerize 
our requirements models and therefore generate fest cases automatically. 


a. Computer implementation 


We use the Visual C++ tool? and the C++ language to develop the Test Case Generation 


tool. First, we perform a global design of the tool using the UML” language, then we generate 
automatically the Visual C++ code-skeleton of the developed UML model and finally, we 
develop the body of the code-skeleton. 


We use the UML editor of Rational Rose (Rational Rose Modeler tool”) in order to perform a 


global design of the Test Case Generation tool. À simplified class diagram With all the 
classes of the tool is shown in Figure 9.5. Two groups of classes are identified. The first one 
is related to the design of the requirements model. The second one deals with the generation 
of test cases. The detailed diagram with the attributes and methods of all the classes and the 
types of relations between classes is not presented here for confidential reasons. 


*7 http://msdn.microsoft.com/fr-fr/visualc/default.aspx (Consulter on November 2008) 
% http:/www.uml.org/ (Consulter on November 2008) 
% http:/www-01.ibm.com/software/awdtools/developer/datamodeler/ (Consulter on November 2008) 
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Figure 9.5 — Simplified class diagram of the Test Case Generation tool 
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We use the Rational Rose Professional C++ tool" in order to generate the Visual C++ code- 
skeleton from the developed class diagram. When generating code-skeleton, the tool 
automatically creates the .h and .cpp files. It generates the classes and adds the aftributes to 
them. It also creates the methods With empty bodies. Afterwards, we must go in and add the 
body of the methods. À screenshot of the code-skeleton generated by Rational Rose is 
presented in Figure 9.6. 
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Figure 9.6 — A screenshot of the C++ code-skeleton generated by Rational Rose 


Once the software architecture of the Test Case Generation tool is generated, we start 
developing in C++ language the body of each method. We have developed about 12500 
Lines Of Codes (excluding comments and blank lines). In fact, we have implemented, using a 
computer language, all the models developed in Chapter 5, 6, 7 and &. 


b. List of main functionalities 


The main functionalities of the Test Case Generation tool are: 


Computerize and verify a requirements model 
Generate Nominal “operation matrices” automatically 
Import “operation matrices” 


Set constraints on the input signals of a requirements model and generate 
automatically the corresponding “operation matrix” 


e _ Simulate a requirements model 


0 http//www-01.ibm.com/software/awdtools/developer/rose/visualstudio/support/ (Consulted on November 
2009) 
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e  Generate fest cases automatically 


Each of these functionalities is developed in the next section. 


IV. Main functionalities of the Test Case Generation tool 


In this section, the main functionalities of the Test Case Generation tool are detailed. 
A. Computerize and verify a requirements model 


After sketching “on paper” the requirements model (Cf. Chapter 5 — Section 5), one can 
computerize this model using the Test Case Generation tool. One can also verify the 
correctness of the computerized model by checking automatically the set of integrity rules 
presented in Table 6.1. À screenshot of the tool after computerizing the requirements model 
of the Chapter 5 — Section 5 example is illustrated in Figure 9.7. 
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Figure 9.7 — A screenshot of the tool after computerizing the requirements model of the Chapter 5 — Section 5 example 
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B. Generate Nominal “operation matrices” automatically 


After computerizing the requirements model, one can generate automatically the two Nominal 
“operation matrices” (Nominal 1 and Nominal 2, Cf. Chapter 7 — Section 2). One can also 
customize these matrices by modifying some succession probabilities or inter-operation time 
interval (Cf. Chapter 7 — Section 2). 


A screenshot of the tool after generating the Nominal “operation matrices” of the Chapter 5 — 
Section 5 example is 1llustrated in Figure 9.6. 
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Figure 9.8 — A screenshot of the tool after generating the Nominal “operation matrices” of the Ch. 5 — Section 5 example 
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C. Import ‘operation matrices” 


One can import “operation matrices”. These matrices can be the results of the bugs and test 
cases reuse processes. They can also be designed manually by an inner engineer. A 
screenshot of the tool after importing “operation matrices” is illustrated in Figure 9.9. 
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Figure 9.9 — A screenshot of the tool after importing “operation matrices”? 


D. Set constraints on the input signals of a requirements model and generate 
automatically the corresponding “operation matrix” 


We develop a computer language that experts can use in order to specify their constraints on 
the input signals. Four types of constraints have been proposed in Chapter 8 — Section 4 
(Logical, Conditional, Succession and Timing constraints). The Test Case Generation tool 
analyzes these constraints and generates automatically the corresponding Driver Profile 
“operation matrix”. The generation of test cases from this “operation matrix” fulfills the 
predefined constraints on input signals. An excerpt on how experts can set constraints on the 
input signals of a requirements model is shown in Figure 9.10. 
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» 
Design of constraints 


The “lgnition” signal is Cyclic 


The “lgnition” signal can be 
different from 2 only if 
"Light Combi _Switch"is equal to 0 


Once "Car Locked'is set to 0, the 
“Light Combi _Switch”signal is set 
to 1 with a probability of 0.5 


Once "Car _Locked'is set to 0, the 
“Light _Combi_Switch”signal is set 
to O with a time interval of 
[13000;13000] 


// Constraint definition 
Constrainti(Cyclic); 

// Set the constraint to 
Ignition(Constraint1); 


// Constraint definition 
Constraint2(NEQUAL, 
// Set the constraint to 
Ignition(Constraint2); 


// Constraint definition 


an input signal 


2, "Light_Combi_Switch", EQUAL, 0); 
an input signal 


Constraint3(1,"Car_Locked", O, 50); 


/ Set the constraint to 


an input signal 


Light_Combi_Switch(Constraint3); 


// Constraint definition 


Constraint4(0,"Car_Locked", 0, 13000, 13000); 


// Set the constraint to 


an input signal 


Light_Combi_Switch(Constraint4); 
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Test Case generation tool 
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Figure 9.10 — An excerpt on how experts can set constraints on the input signals of a 


requirements 


E. Simulate a requirements model 


model 


Once the requirements model is computerized, one can simulate it. Modeler has to define a 
simulation period (the cycle time of the Clock signal, Cf. Chapter 5 — Section 3). Modeler has 
also to specify the path where the simulation plan 1s stored. In fact, a simulation plan consists 
of a finite number of steps. In each step, at most one operation on the input signals is 
performed and an inter-operation time is defined. The result of a simulation is the behavior of 
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the output signals of the requirements model after each step of the simulation plan. The 
output data of a model simulation are stored in an Excel file. In Figure 9.11, we illustrate the 
simulation parameters and the four modes of simulating a requirements model. The “non- 
stop” mode aims to simulate the whole simulation plan nonstop. The “step by step” mode 
consists of simulating the simulation plan in à step by step manner. Indeed, after each step’s 
simulation, the simulation process is stopped. The “period by period” mode stops the 
simulation at each period of the Clock signal. Finally, the “feature by feature” mode stops the 
simulation after each feature simulation. 


|Fle Edit View Help . 
IDSRS{me|SIm ol! mlslw|s 


A"feature by feature” simulation 


cix Cycle time of the Clocksignal 
1 
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Figure 9.11 - The simulation toolbox of the Test Case Generation tool 
F. Generate test cases automatically 


The main functionality of the Test Case Generation tool is to generate fest cases 
automatically. In Chapter 7 — Section 4, we developed a set of test generation objectives and 
constraints. À panel interface to allow the test engineer to set precise targets on these 
objectives and constraints is presented in Figure 7.13. In Chapter 7 — Section 6, we have 
developed a heuristic algorithm to optimize the generation of test cases while fulfilling the 
predefined objectives and constraints. A list of 8 parameters that a test engineer should set 
before start generating rest cases is also introduced. 


Through the Test Case Generation tool, one can set targets on the test generation objectives 
and constraints and parameterize the test generation algorithm. The generation of test cases 1s 
automatic. Each generated test case and its reached objectives and constraints are stored in an 
Excel file. In Figure 9.12, we illustrate the panels where a test engineer can calibrate the 
generation of test cases. We also identify the four modes of generating rest cases. The “non- 
stop” mode aims to generate the set of required test cases nonstop. The “step by step” mode 
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consists of generating each fesf case in a test step by test step manner. Indeed, after each test 
step generation, the test generating process is stopped. When designing a fest step and after 
choosing an operation and an inter-operation time, the “period by period” mode stops the 
model simulation (in order to assess the expected outputs) at each period of the Clock signal. 
Finally, the “feature by feature” mode stops the simulation after each feature simulation. We 
are conscious of the number of parameters (coverage objectives, constraints, optimization 
parameters ...) required to set our approach. In Chapter 10 — Section 8, We propose two 
strategies to help test engineers parameterizing the generation of fest cases. 


Generate the testcases fulfilling the 
predefined objectives and constraints 


Generate “stepby step’the test cases fulfilling 
the predefined objectives and constraints 


After choosing a new operation and inter-operation 
time, simulate the model “period by period” 


After choosing a new operation and inter-operation 
time, simulate the model “feature by feature” 
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Figure 9.12 - The test generation toolbox of the Test Case Generation tool 


Let us consider the example of the Chapter 5 — Section 5. After computerizing and verifying 
the requirements model (Cf. Figure 9.7), one decides to generate the Nominal 1 and 2 
“operation matrices” (Cf. Figure 9.8). For a specific “Configuration” of the functionality 
“Auto_Light” under test (Parameter 6 = 0, Cf. Chapter 7 — Section 6), one decides to generate 
test cases that covers at 100% the domain of all the input, output and intermediate signals 
(coverage objectives, Cf. Chapter 7 — Section 4 and 5). Nevertheless, the length of these rest 
cases must not exceed 50 test steps (time and cost constraints, Cf. Chapter 7 — Section 4 and 
5). The test cases must be generated from the Nominal 2 “operation matrix”. When 
generating the fest cases, one decides to avoid already covered successions of operations 
(Parameter 1 = 1 and Parameter 2 = 30, Cf. Chapter 7 — Section 6). One also decides to 
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optimize the number of test steps by keeping only the ones which contribute to the objectives 
fulfillment (Parameter 3 = 1, Parameter 4 = 10, Cf. Chapter 7 — Section 6). After 10 
generated fest steps With no improvement in the objectives fulfillment, the corresponding test 
case must be ended (Parameter 5 = 10, Cf. Chapter 7 — Section 6). And finally, one decides to 
generate 5 separate fest cases in order to choose the “optimal” one (Parameter 8 = 5, Cf. 
Chapter 7 — Section 6). Since Parameter 6 is equal to 0, Parameter 7 has not to be defined 
(Cf. Chapter 7 — Section 6.B). A screenshot of the Test Case Generation tool after generating 
the test cases for the previous exercise 1s presented in Figure 9.13. The generated fest cases 
and their reached objectives are stored in an Excel file. In case the execution of the fest cases 
on the software product under test is automatic, the generated test cases can be translated into 
a computer language understandable by the test execution platform (Cf. Appendix C). 
Moreover, while simulating a simulation plan on a requirements model or generating test 
cases from a requirements model, one can visualize in real time the covered zones of the 
model (Cf. Chapter 7 — Section 4.B). In fact, the Test Case Generation tool highlights the 
covered zone of the model (Signals domain, Conditions of Decision Tables, States, 
Transitions and Conditions of Finite State Machines and Operation matrices). After 
generating a set of fest cases from the computerized requirements model presented in Figure 
9.7 the covered zones of this model are illustrated in Figure 9.14. 
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Figure 9.13 — A screenshot of the tool after generating test cases for the Chapter 5 — Section 5 example 
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7. Test Case generation tool 
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Figure 9.14 — A screenshot of the tool while highlighting the covered zones of a requirements model 
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V. Conclusion 


The prototype presented in this chapter takes into account the impacts of our approach on the 
processes, roles and tools of the software testing skill within the Johnson Controls 
organization. À new process map for generating automatically test cases from the functional 
requirements of a functionality is presented. New roles and skills for software engineers in 
charge of designing test cases using our approach are developed. And finally, computer tools 
automating up to 70% of our approach are described. The development of these tools 1s not 
presently entirely completed. Some improvements can be done and especially on the 
Graphical User Interfaces. 


In the following chapter, we analyze the results of using this prototype on two industrial case 
studies of practical size. We model, simulate and generate test cases for two software 
functionalities of a car. 
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CHAPTER 10. MODELING AND SIMULATING 
TWO INDUSTRIAL CASE STUDIES 
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I. Introduction 


In order to validate our integrated framework to generate fest cases automatically for a 
software module or product, we consider, at Johnson Controls, two case studies with 
historical data. Through these case studies, we highlight the benefits of using our approach in 
the unit test of software modules. In each case study, we consider one functionality that has 
already been developed and validated in the past using the software Verification and 
Validation (V&V) techniques currently used in Johnson Controls (Cf. Chapter 2 — Section S). 
For each carmaker delivery (Cf. Chapter 2 — Section 3), historical data on the time spent to 
verify and validate these functionalities and on the bugs’ detection by Johnson Controls and 
by the carmakers are available. We consider the first version of the two software modules 
(corresponding to the two functionalities) as it was delivered for the first time by the 
development team to the validation team. We also consider the version of the software 
functional requirements of these functionalities when delivering the software modules for the 
first time to the carmaker. We model, verify, validate and simulate the software functional 
requirements and then generate test cases automatically for the unit test of each software 
module. These test cases are executed on the first version of the two modules. 


Our process to choose the two functionalities under experiment is described in Section 2. A 
characterization of the carmakers’ requirements related to the software of these two 
functionalities is performed in Section 3. Modeling, verification and validation activities of 
the requirements models are respectively presented in Section 4, 5 and 6. A set of “operation 
matrices” for each functionality is designed in Section 7. Strategies to tune the generation of 
test cases are developed in Section 8. The generation and execution of test cases for the two 
functionalities are specified in Section 9. Finally, a deep analysis of the execution results 1s 
illustrated in Section 10. 


IL. Characterization of the two case studies 


We experiment our proposals on two software functionalities of automotive electronic 
products developed within Johnson Controls. The choice of these functionalities has been 
delicate. Many criteria have guided our choices: 


C1: Recent products 

C2: Two different products 

C3: Two different carmakers 

C4: Two different management teams 

CS: Two different development teams 

C6: Two different validation teams 

C7: Two different levels of complexity (from experts point of view) 
C8: Two different types of software functional requirements 

C9: Functionalities already verified and validated using the tradition process 
C10: Functionalities well documented (historical data) 

C11: Functionalities’ experts still in the company (historical data) 


Based on these criteria, we choose two functionalities. The first one is the “front wiper 
management’ functionality. This functionality 1s implemented with other functionalities in an 
automotive electronic product, named body controller module. The second one is the “fuel 
gauge management’ functionality. It is implemented with other functionalities in another 
automotive electronic product, named dashboard or cluster. The compliance of the chosen 
functionalities with the predefined criteria 1s illustrated in Table 10.1. 
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: FER Front wiper Fuel gauge 
SERRE functionality functionality 


Projectstartsin 2005.  Projectstarts in 2005. 


C1 Startof serial Start of serial 
production in 2007 production in 2007 
c2 Body controller module CarDashboardor 
of a car Cluster 
C3 Same carmaker 
Ca Two differentmanagementteams 
C5 Two differentdevelopmentteams 
C6 Two different validationteams 
C7 Quite complex Very complex 
C8 Formal Informal 
ca Fu nctionalities already verified and validated 
using the traditional process 
C10 Historical data are available 
cit Atleast, one expertofthese functionalitiesis still 


in the company 
Table 10.1 - Criteria for selecting the two functionalities 


These two functionalities have already been developed and validated using the Johnson 
Controls present process. Some characteristics of the two software modules developed 
respectively for these two functionalities are given in Table 10.2. 


EE Front wiper functionality Fuel gauge functionality 


Number of input signals of 18 35 
the software module 


Number of output signals 


of the software module 9 25 


Size of the software module 
(Lines Of Code - without 1229 1500 
comments and blanks) 


Table 10.2 - Characteristics of the two software modules developed respectively for the 
two functionalities under experiment 


Each software module delivered to the validation team (first version) is considered to be 
verified and validated independently from its environment (other software modules). This 
means that a code review, a static and dynamic analysis and a unit test (Cf. Chapter 2 — 
Section 5) have been performed on each module delivered to the validation team. 
Unfortunately, at Johnson Controls, bugs detected during these V&V phases are often not 
capitalized in the problems’ database (Cf. Chapter 2 — Section 7.A). Once a bug is detected, it 
is corrected immediately by the person who detects it. Moreover, in Johnson Controls, these 
phases mainly focus on answering the question: “Are we building the product RIGHT?” and 
not on: “Are we building the RIGHT product”. In other words, the compliance with the 
carmaker requirements are not verified on the software modules delivered to the validation 
team. Presently, when testing unitarily a software module, the main purpose of a test engineer 
is to cover at 100% the source code of the module (Cf. Diagnosis 8). The validation team 
integrates the set of modules (already tested unitarily) planned for the carmaker delivery and 
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performs validation tests (Cf. Chapter 2 — Section 5.C and 5.D). During the validation test, 
test engineers design manually test cases in order to demonstrate the compliance of the whole 
software product (integration of at least two software modules) with the carmaker 
requirements (Cf. Chapter 2 — Section 6). Bugs detected by inner engineers during the 
validate test stage (before the delivery) and by the carmaker engineers (after the delivery) are 
capitalized. The distributions of bugs related to the internal behavior of the two functionalities 
are illustrated in Figure 10.1. Until the last carmaker delivery, 22 bugs were detected on the 
software module of the front wiper functionality and 23 bugs on the one of the fuel gauge 
functionality. These bugs were detected, on the two functionalities, before (validation test) 
and after (carmaker test) the carmaker deliveries. Unfortunately, we do not have any 
information on the bugs detected during the other Johnson Controls V&V activities (code 
review, unit test .…..). In fact, after analyzing the total number of the bugs of Figure 10.1, we 
came up to the conclusion that all these bugs could be detected earlier in the process (during 
the unit test stage). In fact, during the unit test of a software module and in addition to a 100% 
code coverage, test engineers should verify the compliance of each software module with the 
carmaker requirements. We call this the functional unit test. In order to comment the Figure 
10.1, let us consider the front wiper functionality example. 17 bugs were detected by the 
Johnson Controls validation test and 5 bugs by the carmaker after intermediate delivery. It 
must be noted that, after developing the software module of the front wiper functionality for 
the first time, only 12 bugs were detected during the first validation stage. Therefore, a 
delivery was performed and the carmaker immediately detected 2 more bugs. In the meantime 
and before the second carmaker delivery, test engineers tried to improve their test cases and 
design some new fest cases. In consequence, they have been able to detect one more bug and 
after the second intermediate carmaker delivery, no new bug was detected by the carmaker. 
For the 4° intermediate delivery, no new fest cases have been developed. The complete 
scenario of bugs’ detection until the last carmaker delivery of the two functionalities 1s 
summarized in the histogram of Figure 10.1. 


Allthese bugs could be detected (earlier in the process) during 
the UNIT test of each software module of the two functionalities 
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Among the bugs detected on the two functionalities, some of them are considered to be more 
critical than others. Severity and Occurrence are two attributes of the current bug model and 
are filled at 99% for each capitalized bug (Cf. Chapter 2 — Section 7.B). These attributes are 
not free fields. Indeed, a set of predefined values for each attribute has been defined by 
Johnson Controls software experts (Cf. Table 10.3). 


Quality of the design of test cases for automotive software: design platform and testing process 


247 


Modeling and simulating two industrial case studies R. AWEDIKIAN 


RE TEE 


Secondary — cosmetic failure, not customer relevant Once — low probability, unlikely failure 

Minor — cosmetic failure, customer relevant Very Rare — low probability, few failures 
Major — workaround exists Rare - moderate probability, occasional failures 
Critical — no workaround exists Often — high probability, repeated failures 


Catastrophic — system crash of the vehicle system (risk of person injury)  Systematic — failure unavoidable 


Table 10.3 - Severity and Occurrence levels as it was defined by Johnson Controls 
software experts (Johnson Controls source) 


Despite the predefined values and according to experts, the attribution of a severity and an 
occurrence for a bug detected internally remains a subjective question. In fact, most of the test 
engineers do not have a global view of the system in order to assess the impact of the detected 
bug on the end-user. However, the severity and occurrence of bugs detected by the carmakers 
are more relevant since they are set by the carmaker itself. The distribution of bugs, detected 
on the two functionalities, across the couple (Severity, Occurrence) is presented in Figure 
10.2. For the front wiper functionality, up to 76% of the bugs are (Minor, Systematic) and for 
the fuel gauge functionality, up to 72% of the bugs are (Major, Systematic). These results 
could be explained from two different points of view. The first one confirms the notion of 
subjectivity in defining a criticity level for a bug. In fact, the bugs of these two functionalities 
were described by two different teams in two different countries. The second one is related to 
the fact that the functionality of managing the fuel level in a car is more critical than the one 
managing the wipers. As a consequence, bugs on the fuel gauge functionality are considered 
to be more critical than the ones of the front wiper functionality. Moreover, bugs detected by 
the carmakers are often considered as critical. 
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Criticity growth 5 Criticity growth 5 


Figure 10.2 - Distribution across the couple (Severity, Occurrence) of the bugs detected 
on the two functionalities 


In Figure 10.3, we illustrate the time spent by the project team in order to debug the software 
modules of the two functionalities using the conventional testing techniques (unit test and 
validation test, Cf. Chapter 2 — Section 5). The main activities done are: 


e Design and execute test cases for the unit test of each software module (Unit test). 

e Analyze the carmaker requirements in order to design validation test cases. 

e Design and execute test cases for the validation test of each functionality. 

e Manage the bugs detected internally and by the carmaker. 

We note that up to 50% and 10% of the total time spent in verifying and validating a software 
functionality were respectively spent to manually design the fest cases and manage the bugs 
detected by the carmakers. Using the current Johnson Controls testing practices, 
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approximately 54 eight-hour days were spent to test the front wiper functionality and 50 


eight-hour days for the fuel gauge functionality. 
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Figure 10.3 - An estimate of the time spent during each delivery to test the two 
functionalities using the conventional testing techniques 


In our experiment, we propose to perform a functional unit test on the software modules 
of the two functionalities. In other words, we plan to verify unitarily the compliance of 
each software module with its functional requirements. To do this, we use our new 
approach to design test cases (Cf. Chapter 5, 6, 7,8 and 9). 
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III. Characteristics of the software functional requirements of the two 
functionalities 


The core of our approach is the modeling of software functional requirements. Therefore, one 
important criterion while choosing the functionalities of the two case studies was the diversity 
of the software functional requirements. In fact, we want to prove that whatever the formalism 
used by the carmaker to express the requirements related to software, one can use our 
approach to generate test cases automatically. In Chapter 2 — Section 4.A, we present the 
result of a study that we performed on the diversity, typology and evolution of these 
requirements within Johnson Controls. Moreover, in Chapter 4 — Section 5.C, we identify 
three formalisms of software functional requirements (/nformal, Semi-formal and Formal). 
Some characteristics of the software functional requirements of the chosen functionalities are 
presented in Table 10.4. 


DRE Front wiper functionality Fuel gauge functionality 


Formalism Formal Informal 
(Cf. Chapter 4 — Section 4.D.1) (Statecharit) (Natural language specifications) 


Size of the software functional 
requirements document 
(Number of pages in Microsoft 
Word format) 


30 30 


Table 10.4 - Characteristics of the software functional requirements of the two 
functionalities 


IV. Modeling the software functional requirements of the two 
functionalities 


Three stages have been necessary for modeling the software functional requirements. The first 
one consists of analyzing and understanding the requirements with our modeling language. A 
loop process was initiated with inner experts in order to well understand and clarify the 
requirements. The second one consists of sketching “on paper” the requirements models. We 
identify the input, output and intermediate signals and the elements (Decision Tables or Finite 
State Machines) of each functionality. Then, we develop each element by identifying all the 
states, transitions and conditions of the elements (Cf. Chapter 5). The third and last stage has 
been the computerization of the requirements models using the Test Case Generation tool that 
we developed (Cf. Chapter 9 — Section 4.A). À comparison of the time spent in each of these 
stages for the two functionalities is illustrated in Table 10.5. 


Time in eight-hour days Front wiper functionality | Fuel gauge functionality 


Time spent to analyze the requirements 


3 3 
before starting modeling task 
Time spent to design “on paper” the 5 7 
requirements model 
Time spent to computerize the paper 12 6 
requirements model 
TOTAL 20 16 


Table 10.5 - Time spent to design the requirements model of the two functionalities 
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In fact, it was more difficult and time consuming to design the requirements of the second 
case study (7 eight-hour days) than the first one (5 eight-hour days). The main reason is that 
the requirements of the second case study are expressed informally. However, we spent more 
üme in computerizing the first case study (12 eight-hour days) than the second one (6 eight- 
hour days). Indeed, the requirements model of the first case study is bigger than the second 
one in terms of number of signals, elements, states, transitions and conditions. In Table 10.6, 
we illustrate the characteristics of the requirements models of the two case studies. The 
requirements model of the “front wiper” functionality has 19 Decision Tables and S Finite 
State Machines, While the one of the “fuel gauge” functionality has 2 Decision Tables and 4 
Finite State Machines. 


| Front wiper functionality Fuel gauge functionality 


# of input signals 18 6 
# of output signals 9 8 
# of intermediate signals 24 31 
# of Decision Tables 1® 2 
# of Conditions in DT 289 110 
# of Finite State Machines 5 4 
# of States in FSM 36 53 
# of Transitions in FSM 119 158 
# of Conditions in FSM 154 197 


Table 10.6 - Characteristics of the requirements models of the two functionalities 
Verifying the requirements models of the two functionalities 


In Chapter 6 — Section 4, We developed a set of integrity rules to be checked on each 
requirements model in order to verify its correctness. The verification of the requirements 
models of the two functionalities was performed manually and automatically. In fact, when 
sketching “on paper” the requirements model, we verify manually the fulfillment of the 
integrity rules. Moreover and after computerizing the model, the Test Case Generation tool 
(Cf. Chapter 9 — Section 4.A) allows to check automatically these rules. Therefore, the time 
spent in verifying these models is integrated to the time of sketching “on paper” and 
computerizing the models (Cf. Table 10.5). We stop verifying a requirements model when all 
the integrity rules are checked OK on the model. After verifying the developed requirements 
models (manually and automatically), around 30 rules violations are detected on each model. 
The distribution of these violations over the set of integrity rules is presented in Figure 10.4. 
We first model the “front wiper” functionality then the “fuel gauge” one. As a consequence, 
violations in Rules 1, 14, 15 and 178 were detected on the first case study and not on the 
second. In fact, when modeling the “fuel gauge” functionality, we focus on respecting the 
rules already violated on the previous case study. Moreover, up to 80% of the violations on 
the first case study (Rule 8) are related to the fact that the domains of the functionality’s input, 
output and intermediate signals are not covered by conditions and actions in elements. Since 
the requirements model of the second case study is smaller than the one of the first case study 
(Cf. Table 10.6), it capitalizes up to 60% of Rule 8 violation. The remaining 40% is shared out 
between the Rules 4, 5, 6 and 7. This is due to the fact that the requirements of the “fuel 
gauge” functionality are expressed informally (Natural language). 
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Figure 10.4 - Distribution of violations over the integrity rules 


VI. Validating the requirements models of the two functionalities 


Once the requirements models are designed, computerized and verified, we validate these 
models. In other words, we verify that the developed requirements models are compliant with 
the carmaker requirements related to the software domain. In Chapter 6 — Section 5, we 
propose three scenarios to validate a requirements model: 


e First scenario: Animate our requirements model 

e _ Second scenario: Simulate test cases delivered by the carmaker on our requirements 
model 

e Third scenario: Compare our requirements model to another valid model of the 
requirements 


These scenarios can be used concurrently or separately. However, it is mandatory to have the 
data necessary for performing one scenario or another. For instance, in case of the first case 
study, the carmaker has delivered a simulation model of the software functional requirements 
and one test case (about 10000 fest steps). Therefore, the three scenarios are applicable. On 
the contrary, the carmaker requirements of the second case study cannot be simulated and no 
carmaker test cases are available. For that reason, only the first scenario can be applied in the 
second case study. Because of time constraints, we only apply the first and second scenarios 
for the first case study and the first scenario for the second case study. One main question is: 
When to stop validating a model? Xn fact, we consider a tradeoff between the quality of the 
model and the resources (time, people, and cost) spent in validation (Cf. Chapter 6 — Section 
2.B). On the one hand, we spent 5 eight-hour days validating the requirements model of the 
first case study and we detect around 15 nonconformities between the model and the 
requirements as 1t was delivered by the carmaker. On the other hand, we spent 20 eight-hour 
days validating the second case study and we detect around 50 nonconformities. Even if the 
requirements model of the first case study is bigger than the one of the second case study (Cf. 
Table 10.6), we spent more time and detected more nonconformities in validating the second 
case study. Indeed, the main reason of this result 1s that the requirements delivered by the 
carmaker for the second case study are informal, while the ones for the first case study are 
formal. In the following, we detail the validation process of the two requirements models. 
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In case of the first case study, we first simulate on our requirements model the test case (about 
10000 test steps) delivered by the carmaker. Once a nonconformity is detected, the 
requirements model is corrected before restarting the simulation of the fest case. The 
cumulated number of nonconformities detected on the first case study is presented in Figure 
10.5. After the 2000" test step, no more nonconformities are detected on the model. 
Afterwards and in order to increase the confidence in our model, we propose to animate it by 
an expert. Two simulation plans of 100 steps each (operations and inter-operation times) 
have been designed by an expert and simulated “step by step” on the model (Cf. Chapter 8 — 
Section 4.E). No nonconformities have been detected. In consequence, we decide to stop 
validating the model. 
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Figure 10.5 —- Cumulated number of nonconformities on the first case study (Second 
scenario) 


In case of the second case study, We animate the model by an expert. Three simulation plans 
of 300 steps each (operations and inter-operation times) have been designed and simulated 
successively “step by step” on the model (Cf. Chapter 8 — Section 4.E). Once a 
nonconformity 1s detected, the requirements model is corrected before restarting the 
simulation. The cumulated number of nonconformities detected on the second case study is 
presented in Figure 10.6. Through the first set of 300 steps, we detect and correct 27 
nonconformities. The second one allows to detect and correct 14 nonconformities. The third 
one detects 8 nonconformities. The main question was: Are there other nonconformities in the 
model? To answer this question, we design a new simulation plan of 50 steps and we simulate 
it on the model. In fact, this simulation plan allows to detect new nonconformities. At this 
moment, we realize the difficulty of validating at 100% a model and we decide to consider a 
tradeoff between the quality of the model and the time spent in validation. In fact, through the 
three simulation plans, We spent up to 20 eight-hour days simulating and debugging our 
requirements model and we cover at 90% the requirements model (signals and elements). 
Therefore, we decide to stop validating the model. 
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Figure 10.6 - Cumulated number of nonconformities on the second case study (First 
scenario) 


Designing “operation matrices” for the two case studies 


The generation of test cases is performed based on the concept of “operation matrix” (Cf. 
Chapter 7 — Section 2). Through an “operation matrix”, inner engineers can enrich the 
requirements model with knowledge on the user (driver) recurrent operations (Cf. Chapter 8 — 
Section 4) and the test engineers’ experience (Cf. Chapter 8 — Section 5 and 6). However, one 
major question 1s: How engineers can design an “operation matrix”? Five possible scenarios 
are identified (Cf. Chapter 9 — Section 3 and 4): 


1. Design manually one or more “operation matrices” and export them to the Test Case 
Generation tool. 

2. Generate the Nominal “operation matrices” (Nominal 1 and Nominal 2) automatically 
via the Test Case Generation tool. 

3. Design manually a set of constraints on the input signals of the requirements model, 
export them to the Test Case Generation tool and generate a Driver profile “operation 
matrix” automatically. 

4. Generate via the Bugs Reuse tool one or more Bug “operation matrices” from one or 
more capitalized bugs and export them to the Test Case Generation tool. 

5. Generate via the Test Cases Reuse tool one or more Test Case “operation matrices” 
from one or more capitalized fest cases and export them to the Test Case Generation 
tool. 


The number of cases of an “operation matrix” for the “front wiper”’ functionality is 9604 
(98x98, 98 is the number of possible operations on the functionality). 7921 (89x89) is the one 
for the “fuel gauge” functionality. Therefore, it was ridiculous to think of manually designing 
“operation matrices” for these functionalities (Cf. Chapter 7 — Section 2). 


As a basic solution, we generate, via the Test Case Generation tool, the two Nominal 
“operation matrices” for the two functionalities (Cf. Chapter 7 — Section 2, Cf. Chapter 9 — 
Section 4.B). According to experts, we define one standard time interval and we affect it to all 
successive operations. 
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Moreover, experts design manually à set of constraints on the input signals of each 
functionality (Cf. Chapter 8 — Section 4) that we export to the Test Case Generation tool. 
Based on these constraints, the Test Case Generation tool generates a Driver Profile 
“operation matrix” for each functionality (Cf. Chapter 9 — Section 4.D). The number and 
type of the designed constraints is illustrated in Table 10.7. 


Type of constraints : ; à , ; 
(Cf. Chapter 8 - Section 4) Front wiper functionality Fuel gauge functionality 


Logical 


1 
Conditional 3 
Succession 2 

1 


— 


Timing 
Table 10.7 - Constraints designed for the two functionalities 


Unfortunately, we do not have enough time to analyze bugs and fest cases capitalized on 
previous projects implementing the “fuel gauge” functionality. In fact, we decide to focus our 
effort on the “front wiper” functionality. In Chapter 2 — Section 7.C, we perform a study on 
the bugs detected on the “front wiper” functionality through 5 different projects since 1997 
and till 2007. Excluding the last project (Project 5) which is the one on which we carry out 
our experiments, 55 bugs were detected on this functionality since 1997. In Chapter 8 — 
Section 5, We propose two strategies to reuse capitalized bugs. One strategy (Cf. Chapter 8 — 
Section 5.A) consists of representing the “Problem description” of bugs in a specific format in 
order to generate a Bug “operation matrix” for each bug. One difficult task was to represent 
the “Problem description” of the 55 identified bugs into our specific format. Based on the 
experts’ advices, we only consider the 10 most critical bugs providing that there exists enough 
information related to the “Problem description” of the bug. Afterwards, we generate, via the 
Bugs Reuse tool (Cf. Chapter 9 — Section 3.C.I1), the corresponding 10 Bug ‘operation 
matrices” that we export to the Test Case Generation tool. À glossary of the input signals 
names on the previous and current projects was necessary. 


Over the 4 projects implementing the “front wiper” functionality (Cf. Chapter 2 — Section 
7.0), only one project P has adopted the fest case format presented in Definition 2.11. This 
format of rest cases is required for generating a Test Case “operation matrix” automatically 
for each test case (Cf. Chapter 8 — Section 6). Within P, test engineers have designed one test 
case (about 2000 test steps) in order to test the “front wiper” functionality. Based on this test 
case, We generate, via the Test Cases Reuse tool (Cf. Chapter 9 — Section 3.C.2), one Test 
Case ‘operation matrix” that we export to the Test Case Generation tool. À glossary of the 
input signals names on the previous and current projects was necessary. 


A summary of the “operation matrices” designed for the two functionalities is illustrated in 
Table 10.8. We also estimate the time spent in designing these “operation matrices”. For the 
front wiper functionality, we spent 2 eight-hour days and for the fuel gauge functionality, 0,5 
eight-hour days. …n fact, identifying and preparing the capitalized bugs and fesf cases have 
taken about 1,5 eight-hour days. 
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Front wiper functionality Fuel gauge functionality 


2 Nominal « operation matrices » 2 Nominal « operation matrices » 
(Nominal 1 and Nominal 2) (Nominal 1 and Nominal 2) 
1 Driver Profile « operation matrix » 1 Driver Profile « operation matrix » 


10 Bug « operation matrices » - 


1 Test Case « operation matrix » - 


Table 10.8 — “Operation matrices” designed for the two functionalities 


VIIL. How to tune the generation of test cases? 


Three questions have been raised at this stage of the experiment: 


e From which “operation matrix” do we start generating fest cases”? 
e How to tune the coverage objectives and the time and cost constraints? 
e How to tune the test generation algorithm? 


In order to answer the first question, we propose to generate fest cases from the “operation 
matrices” according to the order presented in Figure 10.7. Firstly, We generate fest cases 
from the Bug “operation matrices”. At least, we ensure that our software module is free from 
bugs similar to the ones already detected in the past. Secondly, we generate test cases from the 
Test Case “operation matrices”. These test cases are suitable to detect bugs since they are 
based on the test engineers’ experience. Thirdly, we generate test cases from the Driver 
Profile “operation matrix”. This aims to check that the software module fulfills the end-user 
(driver) expectations. Finally, we generate test cases from the Nominal “operation matrices”. 
Improbable successions of operations are generated in order to check the robustness of the 
software module. In the previous section, we note that no Bug or Test Case “operation 
matrices” have been designed for the second case study. Moreover and according to experts, 
simulating random operations (Nominal “operation matrices”) on the fuel gauge 
functionality does not make real sense. Therefore, for this case study, we only generate test 
cases from the Driver Profile “operation matrix”. 
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Figure 10.7 — Our strategy of generating test cases from the “operation matrices”? 


Before generating test cases from an “operation matrix”, We have to define the objectives and 
constraints of the generation (Cf. Chapter 7 — Section 5, Cf. Chapter 9 — Section 4.F) but also 
the parameters of the optimization algorithm (Cf. Chapter 7 — Section 6, Cf. Chapter 9 — 
Section 4.F). In our case studies, we tune these factors based on a fry-and-test protocol and on 
the experts’ knowledge. 
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According to the type of the “operation matrix”, We propose guidelines for defining the 
coverage objectives and the time and cost constraints (Cf. Table 10.9). These guidelines have 
been defined based on our analysis of the different “operation matrix” modes. For instance, 
in case of a Bug or Test Case “operation matrix” mode, the knowledge extracted from 
capitalized bugs or test cases is incorporated in the “operation matrix”. Therefore, it 1s 
necessary to cover at least all the successions of operations of a Bug or Test Case “operation 
matrix”. The constraints’ values depend on the context (budget, planning, and resources) of 
the project. 


“Operation matrix” from which one AE ; SRE 
the test case has to be generated Objectives guidelines Constraints guidelines 


, À At least, cover at 100% the « operation 
Bug « operation matrix » 


matrix » 
E The number of test steps and the 
; : A least, cover at 100% the « operation one 
Test Case « operation matrix » tee execution time of the generated 
à test case depend on the context 
: : : : Aleast, cover at 100% the domains of ; 
Driver Profile « operation matrix » (budget, planning, resources) of 


the input signals 
At most, cover the requirements model 
andthe « operation matrix » 


the project 
Nominal « operation matrix » 


Table 10.9 - Guidelines for defining the objectives and constraints of a test case 
generation 


Based on these guidelines, we set the objectives and constraints of the fest cases generation 
for the two functionalities (Cf. Table 10.10). Because of technical reasons and based on the 
assumption that covering at 100% the requirements model involves that the source code is 
covered at 100%, we decide to only set objectives in terms of functional coverage. From the 
opposite direction, this assumption is rarely true (Cf. Chapter 7 — Section 4). Moreover, we do 
not consider the criticity of the requirements (Critical Successive 2-Operations coverage, DT 
Critical Conditions coverage and so on — Cf. Chapter 7 — Section 4.B). Finally, no constraints 
are set in terms of number of test steps and execution time of the generated test cases. 


Front wiper functionality Fuel gauge functiona!ity 
“Operation matrices” \ TestCase, Nominal? ; 
Objectives Fa Fa 
Functional coverage . ; ni 
Inputs domains - - 100-10 100-10 À - 100-10 + 
Outputs domains - - - 100-10 à ÿ - : cs. 
Intermediates domains - - - 100-10 + 'E - \ - j 
Inputs boundaries - - - 100-10 - : ï - - à - : 
Outputs boundaries -  Target- Weight - 100-10 - \ Î - - ï ; 
Intermediates boundaries - - - 100-10 ms. Ni & - ÿ 
Successive 2-Operations 100-10 100-10 - 100-10 - ii - - À 
DT Conditions ; : ; 100-10 ire : a: 
FSM States - - - 100-10 - {| \- - si 
FSM Transitions - - - 100-10 -/ \. - aa 
FSM Conditions - - - 100-10 “ à : : a 
Constraints : Fu | 
Testexecutiontime and cost Î i 
Test Execution Time (en ms) = - - - ; - = - ' = \ 
Test Step Number - - - - ue - - é - \ 


Table 10.10 — Objectives and constraints when generating test cases for the two 
functionalities 
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Once defining objectives and constraints, we have to tune the optimization algorithm of the 
test case generation. In Chapter 7 — Section 6, We describe the optimization algorithm and its 
parameters. Eight parameters have been identified. In our case studies, we tune these 
parameters based on the traditional fry-and-test protocol. In fact, we first set the objectives 
and constraints of the test case generation, then we set specific values on the optimization 
algorithm parameters and finally we generate test cases. Based respectively on the fulfillment 
and respect of the objectives and constraints, we adjust the optimization parameters. The 
purpose is to better fulfill and respect the coverage objectives and the time and cost 
constraints. For each case study and after 10 trials (approximate), the “optimal” values for the 
optimization algorithm parameters are identified in Table 10.11. We spent 1 eight-hour day in 
adjusting these parameters for the two case studies. Let us consider the first case study. We 
have to generate test cases from a Bug “operation matrix”. According to Table 10.9, we first 
set the objectives and constraints of the fest case generation (Cover at 100% the Bug 
“operation matrix”). Afterwards, we tune the parameters of the optimization algorithm (Cf. 
Table 10.11). In fact, we decide to optimize the coverage of the Bug “operation matrix” 
(Parameter 1 = 1). When choosing a new operation in the “operation matrix”, the 
optimization algorithm checks if the corresponding succession of operations is already 
covered or not. If it 1s the case, another operation is chosen until a non-covered succession of 
operations 1s selected. The maximum number of unsatisfied trials, before the algorithm exits 
the loop, is 30 (Parameter 2 = 30). Even if the designed fest step does not improve the 
objectives fulfillment, we decide to add it to the test case under construction (Parameter 3 = 0 
and Parameter 4 = 0). After 30 test steps generated without an improvement in the objectives 
fulfillment, we decide to stop designing fest steps for the corresponding test case (Parameter 
5 = 30). According to experts, we decide to generate test cases for only one “Configuration” 
of the front wiper functionality (Parameter 6 = 0 and Parameter 7 not defined). In fact, we 
choose the basic (by default in a car) “Configuration” of the functionality. Finally, only one 
test case has to be generated (Parameter 8 = 1). 


Front wiper functionality Fuel gauge functionality 

“Operation matrices” | Bug | TestCase | DriverProfile | Nominal D". [Driver Profile}, Nominal / 

Parameter 1 1 MIE | 

Parameter2 30 < 

Parameter 3 0 

Parameter 4 0 

Parameter5 30 

Parameter6 0 4 x, 

Parameter 7 

Parameter 8 1 6 6 6 Cu s À 3 


Table 10.11 - Optimization parameters when generating test cases for the two 
functionalities 


IX. Generation and execution of the test cases on the software modules of 
the two functionalities 


In Section 7, we develop the “operation matrices” designed for the two case studies. In 
Section 8, objectives, constraints and optimization parameters for the generation of test cases 
for the two functionalities are defined. In this section, we describe the generation and 
execution of fest cases. The number and characteristics of the generated rest cases are 
illustrated in Figure 10.8 and 70.9. In fact, the generation of test cases has been carried out 
automatically via the Test Case Generation tool (Cf. Chapter 9 — Section 4.F). 
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10Bug 1 Driver Profile “operation 
operation matrices” matrix” 


10testcases (1 testcase 6 testcases fromthe 

from each Bug“operation Driver Profile “operation 
matrix”) matrix” 
Eachtestcase, around 10 Eachtestcase, around 
teststeps 1000 teststeps 
Foreachtest case, Foreachtestcase, 
objectives are fulfilled at objectives are fulfilled at 
100% 70% 


eo FRONT eo WIPER eo FUNCTIONALITY 
“operation matrix” “operation matrix” 


6 test cases fromthe Test 6 testcases fromthe 
Case “operation matrix” Nominal2"operation 
Eachtestcase, around matrix 

400 test steps Eachtestcase,around 
Foreachtest case, 10000teststeps 
objectives are fulfilled at Foreachtestcase, 
99% objectives are fulfilled at 
0% 
Figure 10.8 - Order of generating and executing test cases for the front wiper 
functionality 
1 Driver Profile “operation 
matrix” 
3 testcases fromthe 
Driver Profile “operation 
matrix” 
Eachtestcase, around 
300 test steps 
Foreachtestcase, 
objectives are fulfilled at 
90% 


É) FUEL © GAUGE ) FUNCTIONALITY 


Figure 10.9 — Order of generating and executing test cases for the fuel gauge 
functionality 


As we plan to do a functional unit test of the two functionalities (Cf. Section 2), we execute 
the generated fest cases on the first version of the two corresponding software modules (as it 
was delivered for the first time by the development team to the validation team). In other 
words, we isolate the first version of the software module which fulfils the front wiper 
functionality and the one of the fuel gauge functionality and we test them through the 
generated test cases. To do this, we first translate these fesf cases, Via an inner tool, into the 
unit test language. It is a computer language understandable by the unit test execution 
platform (Cf. Appendix B and ©). The execution is performed following the order defined in 
Figure 10.8 and 70.9. Once an anomaly is detected, we analyze it in order to identify its 
origin. The origin can be: 


e Abuginthe requirements model, 

e À known bug in the software module. It is a bug that the validation test of Johnson 
Controls or the carmaker has already detected (Cf. Figure 10.1), 

e An unknown bug in the software module. It is a bug which is not yet detected neither 
by the validation test of Johnson Controls nor by the carmaker. 
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Whatever the origin, the bug is corrected before restarting the execution of the fest cases. 


It is important to note that the time to generate test cases Via the Test Case Generation tool 
and the time to execution fest cases Via the unit test execution platform are both trivial from 
the automotive industry point of view. It can be respectively estimated to 500 and 1000 fes 
steps per minute. These estimations are given for reference only because they depend on 
many factors (CPU“', inter-operations times of the test steps, parameters of the optimization 
algorithm and so on). 


X. Analysis of the results of the two case studies 


A. Detect bugs earlier in the software life cycle 


Once executing all the generated test cases on the software modules of the two functionalities, 
a total of 29 anomalies were detected on the first case study and 35 anomalies on the second 
one. In fact, it is important to assess the accuracy of the results delivered by our measurement 
system. Therefore, we measure (Cf. Figure 10.10): 


e The ratio between the number of “false” bugs (bugs in the requirements models) 
detected by our approach and the total number of detected anomalies. The “false” 
bugs are the anomalies that are not related to bugs in the software module under test 
but to bugs in the requirements model itself. As said in Section 6, it is impossible to 
validate at 100% a requirements model and therefore bugs in this model could be 
detected later when executing the generated test cases on the software under test. 

e The ratio between the number of “true” bugs (known bugs in the software modules) 
detected by our approach and the total number of detected anomalies. 

e The ratio between the number of “new” bugs (unknown bugs in the software modules) 
detected by our approach and the total number of bugs in the software module under 
test. 


About 17% (5 over 29) of the anomalies detected on the front wiper functionality were related 
to bugs in the requirements model and up to 49% (17 over 35) on the fuel gauge functionality. 
This could be explained by the fact that the requirements models of the two functionalities 
could not be exhaustively validated (Cf. Section 6). More especially, the one of the fuel gauge 
functionality because of the informal formalism of the carmaker requirements. Around 65% 
(19 over 29) of the anomalies detected on the front wiper functionality were related to known 
bugs in the software module and up to 51% (18 over 35) on the fuel gauge functionality. We 
also detect 5 “minor” bugs (“minor” from experts’ point of view) that neither the conventional 
validation test of Johnson Controls nor the carmaker test has detected on the front wiper 
functionality. According to experts, these bugs have no impact on the end-user (driver). It 
represents 19% (5 over (22+5)) of the total number of bugs in the functionality (22+S). 


From another point of view, we were able to detect 86% (19 over 22) of the bugs already 
detected by the conventional validation test on the first case study and 78% (18 over 23) on 
the second one. These results prove that many of the bugs detected later in the software 
life cycle (Validation test) could be detected earlier (Unit test) via our functional unit test. 


*! CPU : Central Processing Unit 
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Figure 10.10 — Origin of the anomalies detected when executing the generated test cases 
on the two functionalities 


After analyzing the remaining 3 (22-19) and 5 (23-18) known bugs not detected respectively 
on the first and second case studies, we come up to the conclusion that these bugs could be 
detected by our platform since we reach a 100% of the functional coverage (which is not the 
case, Cf. Section 9). These non-detected bugs are related to some specific functional 
requirements that weren’t covered by our generated fest cases. Indeed, when generating test 
cases from a Nominal “operation matrix”, our computational algorithms didn’t succeed to 
reach 100% of the functional coverage (maximum of 90%). To overcome this lack, we have 
to improve our computational algorithm in order to focus on covering the non-covered zones 
of the requirements model. In Figure 10.11, we identify across the carmakers’ deliveries the 
known bugs (Cf. Figure 10.1) not detected by our approach. In Figure 10.12, we illustrate the 
criticity (Severity, Occurrence) of the non-detected bugs as it was filled in the problems’ 
database (Cf. Figure 10.2). 
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Figure 10.11 — Distribution according to the carmakers’ deliveries of the known bugs 
not detected by our approach 
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Figure 10.12 - Distribution across the couple (Severity, Occurrence) of the known bugs 
not detected by our approach 


Among the known bugs detected by our approach, some of them are bugs already detected by 
the conventional Johnson Controls validation test and others by the carmaker (Cf. Figure 
10.13). For the front wiper functionality, we detect 60% (3 over 5) of the bugs detected by the 
carmaker and 94% (16 over 17) of the bugs detected by the conventional validation test. For 
the fuel gauge functionality, we detect 80% (4 over 5) of the bugs detected by the carmaker 
and 78% (14 over 18) of the bugs detected by the conventional validation test. 
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Figure 10.13 — Origin of the known bugs detected by our approach on the two 
functionalities 


In the case of the front wiper functionality, the evolution of the cumulated number of known 
and unknown bugs that we detect through our approach is illustrated in Figure 10.14. The 
evolution is drawn according to the execution order of the generated test cases defined in 
Figure 10.8. Once a bug is detected, it is corrected before restarting the execution. Through 
the test cases generated from the Bug “operation matrices”, we detect 2 bugs out of the 17 
bugs detected by the Johnson Controls software testing processes. The rest cases generated 
from the Test Case “operation matrix” enable to detect 6 bugs out of the 17 bugs detected by 
Johnson Controls, 1 bug out of the 5 bugs detected by the carmaker after intermediate 
delivery and 2 new “minor” bugs that were neither detected by Johnson Controls nor by the 
carmaker. The fest cases generated from the Driver Profile “operation matrix” enable to 
detect 1 bug out of the 17 bugs detected by Johnson Controls. And finally, the rest cases 
generated from the Nominal 2 “operation matrix” enable to detect 7 bugs out of the 17 bugs 
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detected by Johnson Controls, 2 bugs out of the 5 bugs detected by the carmaker and 3 new 
“minor” bugs that were neither detected by Johnson Controls nor by the carmaker. As 
conclusions on the bugs’ detection flow: 


e All new detected bugs (5 bugs) have occurred in the Test Case and Nominal 2 test 
stages. This could be explained by the fact that these bugs are related to specific 
successions of operations, illogical from a use point of view but could probably occur 
in the serial life of the software product. 

e _Atthe end of each test stage, the number of the detected bugs tends to stabilize. 
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Figure 10.14 - Evolution of the cumulated number of bugs detected by our approach on 
the front wiper functionality 


We also execute independently on the first version of the front wiper software module all the 
test cases generated from each mode of “operation matrix”. The result of this experiment 1s 
illustrated in Figure 10.15. We identify the number and type of bugs that can be detected by 
one or more modes of “operation matrix”. As a conclusion: 


e One mode of the “operation matrix” Wwasn'’t able to detect all the bugs already 
detected by the present Johnson Controls testing processes and by the carmaker. 
Each mode has at least one bug that can only be detected via this mode. 

e The Nominal 2 mode “operation matrix” detects the maximum number of bugs. This 
could be explained by the fact that we generate 60000 test steps from this “operation 
matrix” and we cover at 90% the requirements model. 
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e The Test Case mode “operation matrix” detects up to 80% of the bugs that the Drive 
Profile mode can detect. In fact, the capitalized test cases have been designed with an 
end-user point of view. 
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Figure 10.15 - Number and type of bugs detected via each ‘operation matrix” mode 


In the case of the fuel gauge functionality, the evolution of the cumulated number of known 
and unknown bugs that we detect through our approach is 1llustrated in Figure 10.16. Through 
the test cases generated from the sole Driver Profile “operation matrix” (Cf. Section 8), we 
detect 14 bugs out of the 18 bugs detected by Johnson Controls and 4 bugs out of the 5 bugs 
detected by the carmaker. No new bugs have been detected. Comparing to the first case study, 
we were able through the Driver Profile mode to detect most of the known bugs. This could 
be explained by two facts: 


e We cover 90% of the input signals domains in comparison with 70% in the first case 
study. 

° _According to experts, simulating random operations (Nominal “operation matrices”) 
on the fuel gauge functionality does not make real sense. 
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Figure 10.16 — Evolution of the cumulated number of bugs detected by our approach on 
the fuel gauge functionality 


B. Decrease the time spent in testing a functionality 


On the one hand, we detect bugs earlier in the software life cycle. On the other hand, we 
lower the time spent in testing a functionality. Thanks to historical data, the total time spent 
in testing conventionally the two functionalities is 1llustrated in Figure 10.17; e.g. 53.75 eight- 
hour days for the front-wiper and 50 days for the fuel gauge. For the front wiper functionality, 
no unit test has been performed. During the validation test stages, test engineers spent 11,5 
eight-hour days analyzing the carmaker requirements before start designing manually test 
cases (29,5 eight-hour days). 6 eight-hour days were spent executing the designed test cases 
and analyzing the results. Finally, we estimate at 6,75 eight-hour days the time spent in 
managing the bugs detected later in the process. For the fuel gauge functionality, 5 eight-hour 
days were spent testing unitarily the functionality. During the validation test stages, test 
engineers spent 10 eight-hour days analyzing the carmaker requirements before start 
designing manually test cases (22 eight-hour days). 6 eight-hour days Were spent executing 
the designed fest cases and analyzing the results. Finally, we estimate at 7 eight-hour days the 
time spent in managing the bugs detected later in the process. 


As stated in Section 2, up to 50% and 10% of the total time spent in verifying and validating a 
software functionality were respectively spent to manually design the test cases and manage 
the bugs detected later in the process. 
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Figure 10.17 — An estimate of the total time spent in testing conventionally the two 
functionalities 


The total time that we spent in testing unitarily the two functionalities using our approach is 
presented in Figure 10.18. It has been approximately spent 39 and 41,5 eight-hour days 
testing respectively the front wiper and fuel gauge functionalities. In Section 4, 5, 6, 7 and 9, 
we estimate and comment the time spent analyzing the carmaker requirements, modeling, 
computerizing, verifying and validating the requirements model, designing the “operation 
matrices” and finally generating and executing automatically the test cases. After executing 
the generated test cases, we estimate to 10 and 2 eight-hour days the time respectively spent 
in analyzing the execution results. It consists, once an anomaly is detected, of answering the 
question: “Is it a bug in the requirements model or a bug in the software module?”. The 
correction of anomalies 1s instantaneous. The time spent in analyzing the execution results 1s 
proportional to the number of executed test steps (front wiper: 68500 test steps, fuel gauge: 
900 test steps). In fact, the task of manually designing the test cases disappears in favor of 
designing, verifying and validating the requirements model. Once the model is developed, the 
test design activity is automated but more efforts are necessary to analyze the results of the 
tests execution. Indeed, test engineers have to understand the generated fest cases in order to 
confirm or not a bug. Moreover, we do not detect 3 and 5 known bugs respectively on the first 
and second case studies. In Section 10.A, we come up to the conclusion that these bugs could 
be detected by our platform since we reach a 100% of the functional coverage (now, it is not 
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the case, Cf. Section 9). Based on the assumption that our computational algorithm was 
improved (to be able to reach the 100% functional coverage), we estimate the time required to 
detect the remaining bugs on the two case studies. This time take into account the time to 
generate and execute the test cases and analyze the results. For the first case study, we already 
cover 90% of the requirements model and 3 bugs are remaining. Therefore, we estimate to 2 
eight-hour days the time to detect these bugs. For the first case study, we already cover 70% 
of the requirements model and 5 bugs are remaining. Therefore, we estimate to 3 eight-hour 
days the time to detect these bugs. These estimations could be explained by the fact that: 


e The requirements model of the first case study is bigger than the one of the second 
case study (Cf. Table 10.6). 

e  Analyzing the execution results of the second case study takes more time that the one 
of the first case study. In fact, the requirements model of the second case study 
(Natural language) is less reliable that the one of the first case study (Cf. Section 3 
and 6). 


Globally, we spent approximately 39 and 41,5 eight-hour days testing respectively the front 
wiper and fuel gauge functionalities. In this estimation, we do not consider the time spent in 
configuring our test platform using the fry-and-test protocol (Cf. Chapter 8.A). In conclusion, 
we lower by 27% (39 instead of 53,75 eight-hour days) and 17% (41,5 instead of 50 eight- 
hour days) respectively the time spent in testing the front wiper and fuel gauge functionalities. 
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Figure 10.18 — An estimate of the total time spent in testing unitarily the two 
functionalities using our approach 
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After the first carmaker delivery and for each new delivery, we estimate that an average of 1 
eight-hour days can be enough to review and update the test cases of the functionality under 
test. In fact, as carmaker requirements is suitable to evolve along the different deliveries (Cf. 
Chapter 2 — Section 4.A), it will be easier to test engineers to update the requirements model 
and generate automatically a new set of test cases than to update manually the design of test 
cases. 


CC: Quantitative results’ overview: earlier detection of bugs and time saving 


Performing a functional unit test, for each functionality (software module), using our 
approach to generate test cases automatically leads to notably improved results. In Table 
10.12, we summarize the results of the two case studies in terms of detecting bugs earlier in 
the software life cycle. 


D | Front wiper functionality | Fuel gauge functionality 


Increase the number of bugs detected 100% (from 12 to 24) 800% (from 2 to 18) 
since the first testing phase 

Decrease the number of bugs detected by 60% (from 5 to 2) 80% (from 5 to 1) 
the carmaker 

Increase the number of bugs detected by 41% (from 17 to 24) 22% (from 18 to 22) 
Johnson Controls 

New bugs detected 18% (5 out of 27) 0% (0 out of 23) 


Table 10.12 — A summary of the results of the two case studies 


Moreover, we lower by 27% and 17% respectively the time spent in testing the front wiper 
and fuel gauge functionalities (Cf. Figure 10.19). 
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Figure 10.19 - Reducing the time spent in testing the two functionalities 


XI. Conclusion 


In this chapter, we have experimented our new testing methodology through two typical case 
studies on historical data. Potential benefits (quantitative and qualitative) have been 
quantified. We reduce by 70% the number of bugs detected by the carmakers and by 9% the 
ones detected by the end-users. Moreover, we reduce by 22% the time spent in testing a 
software product. We also propose to deliver to the carmaker formal quality indicators 
(coverage) on the delivered software. All these results contribute to an improvement of the 
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customer satisfaction and as a direct impact; the number of tenders will grow. Unfortunately, 
estimating the cost of software bugs in an organization is a delicate, strategic and confidential 
question and therefore we were not allowed to communicate the numbers on the bugs’ costs 
savings via the use of our approach. 


In the following chapter, we give an overview on the contributions, impacts and perspectives 
of our approach. 
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I. Contributions’ review 


In this research project, we were asked by an automotive electronic supplier, namely Johnson 
Controls, to improve the performance of its software V&V activities. Their main purpose is to 
improve the quality of their products and therefore better satisfy the requirements and 
expectations of their clients. We went through this problem with a systemic approach in order 
to identify levers in any domains from which we might be able to improve the global 
performance of the software V&V activities. The major added value of the present work is to 
globally solve the quality issue of the software testing process. Hereafter, we summarize the 
main ten contributions of our research: 


Contribution 1: A list of anomalies and lacks in the software verification and validation 
(V&V) practices in automotive industry. 

Through an industrial audit, we analyze the current software practices in automotive industry. 
The audit is divided into four parts: 1) the process of managing the carmakers’ requirements 
related to the software domain, 2) the processes of verifying and validating software products, 
3) the process of managing and reusing capitalized bugs and finally 4) the process of 
managing and reusing capitalized fest cases. For each of these parts, we make our analysis in 
two stages: 1) a snapshot of the current software practices in automotive industry (process, 
tool, people) and 2) an analysis and identification of issues and lacks (diagnoses) in these 
practices. Our approach to perform the audit can be divided into 7 activities: 1) analyze the 
documents delivered by the carmakers to their electronic suppliers, 2) analyze the main 
activities of an engineer When designing fest cases for a software product, 3) audit engineers 
when designing test cases, 4) intervention on the design of test cases for four software 
projects, 5) interview managers on the expectations of the carmakers at each stage of the 
software development life cycle, 6) interview all types of engineers that can be involved in a 
software project and finally 7) analyze data on the software testing practices of carmakers. 
The result of the audit is a list of anomalies and lacks (diagnoses) in the current software V&V 
activities in automotive industry. 


Contribution 2: A formal specification language to represent and simulate software 
functional requirements in automotive industry 

Managing the software functional requirements 1s considered as one of the key issues in the 
software development process. In fact, these requirements are the main input for the design 
and implementation processes of the software product but also for the verification and 
validation processes. Ten years ago, formal methods were rarely used in automotive industry, 
contrarily to medical, avionics and railways industries. Now, in automotive industry, semi- 
formal and formal methods are more and more used to specify software functional 
requirements. However, there 1s a lack of a standard formalism shared between carmakers and 
suppliers. In fact, for each project, the supplier must adapt its processes to the formalism used 
by the carmaker. In this context, we develop a new formal and simulation language to model 
software functional requirements. À simulation model of these requirements can help to avoid 
ambiguity, incompleteness and inconsistency in customers’ requirements. Development and 
validation teams can communicate more easily with the customer and fix specification’s 
problems. Moreover, through a simulation model, one can automate the assessment process of 
all the expected outputs values of a software product. In fact, when designing test cases, test 
engineers can perform the selected operation on the requirements model and automatically 
assess the expected output values by simulating the model. We then name “operation” the fact 
that an input signal of the software product is set to a given value. Finally, one can now 
Jformally measure the coverage of the requirements model, which bring new valuable quality 
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indicators in addition of the sole code coverage for better monitoring the software testing 
process. 


Contribution 3: An automatic process to design test cases for software products 

In industry, the activity of manually designing test cases for software products becomes more 
and more laborious and time consuming. Despite the considerable time and money spent in 
testing a software product and after each delivery to the customer, some bugs are still detected 
by the customer. Since the late 90’s, the automation of the test case design process has 
become a hot topic and industrials are still looking for a relevant automation of this process. 
In this context, we develop a strategy to automatically design test cases With simulations from 
our formal model. A test case is à series of operations Whose selection is performed based on 
a Monte Carlo simulation on an “operation matrix”. Probabilities are expressed for choosing 
a next operation and for defining the time interval between both successive operations. 
Therefore, we build a matrix that we name ‘operation matrix” With all possible operations in 
columns and in rows; this “operation matrix” becomes central to our test case generation 
algorithm. All along the rest case generation, the expected values on the output signals of the 
functionality are assessed through a simulation of the requirements model. 


Contribution 4: An objective function for optimizing the design of test cases for software 
products 

Testing software exhaustively remains a major problem from the computing point of view. 
Therefore, software testing must often be based on specific assumptions and objectives which 
help test engineers and managers to decide when to stop the testing protocol. In order to 
monitor our automatic design of test cases, We propose an objective function based on a 
Jormal structural (software code) and functional (customer requirement specification) 
coverage and the execution time and cost of designed test cases. In software engineering, the 
term “coverage” means the degree, expressed as a percentage, to which a specified item (code 
or requirement) has been exercised by a fest case. In addition, we define an exponential set of 
weights that test engineers can associate for each defined coverage, time or cost target: O (to 
be ignored), 1 (not very important), 5 (important), 10 (very important). 


Contribution 5: A hybrid heuristic algorithm for optimizing the design of test cases for 
software products 

When testing a software product, test engineers have to execute the designed fest cases on the 
software under test. The execution could be manual or automatic and is often time and 
resource consuming. The main purpose of a test engineer 1s to detect the maximum number of 
bugs in minimum laps of time. Therefore, optimizing the number and length of fest cases 
while fulfñlling predefined objectives and constraints 1s critical to reach the quality, schedule 
and cost goals of a software project. To overcome this problem, we propose a heuristic 
algorithm in charge of optimizing the design of test cases while fulfilling quality objectives 
and time constraints. In this algorithm, we implement two types of optimization strategies: 
Look Back and Look Ahead. ]n fact, when designing test cases, We avoid similar and 
repetitive operations or successions of operations (Look Back) and we focus on the ones 
which improve the objective fulfillment (Look Ahead). 


Contribution 6: À software bug classification model and a detailed typology of software 
problems 

Each software organization uses a problems’ tracking tool in order to manage and store 
problems detected during a software project. Moreover, the fracking tool has a database where 
all the problems are stored. Such databases hold thousands of software bugs and are difficult 
to be analyzed. In fact, when describing a bug in the problems’ tracking tool, there are often 
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too many fields to fill in, a lot of free fields and a lack of relevant predefined fields. Moreover 
and as the detection of bugs comes later in the process, engineers do not have enough time to 
fill in all the fields of a bug. Therefore, analyzing these databases in order to pinpoint issues in 
the development processes and propose improvement actions is a complicated task. In this 
context, We propose a new bug classification model. The aim of this model is to be able to 
identify process improvement actions for the development and V&V processes. In other 
words, the new bug classification model answers the question of “which types of software 
problems are injected and detected in which process phase?” To do this, we propose a detailed 
software problem typology taking the industrial context into account. In addition, identifying 
recurrent type of software problems allows test engineers to focus the design of fest cases on 
detecting these problems. 


Contribution 7: À process to define software users’ profiles in order to design test cases that 
simulate the real use of a software product 

There is no better way to test a product other than testing it in the way that it will be used. The 
major number of bugs detected by the end-users of a software product is related to specific 
operations Or successions of operations recurrently performed on the software in real use. 
Therefore, testing a software product with an end-user point of view seems beneficial. We 
propose to define an end-user behavior’s profile for each software under test. This profile can 
be used by test engineers when designing fest cases. In fact, we define four types of 
constraints that test engineers can affect to each input signal of a software product in order to 
eliminate or favor specific successive operations. Each input signal can have one or more 
constraints. These constraints aim to lower the number of possible combinations on input 
signals and to more thoroughly pinpoint which ones are frequently set once the product is 
launched on the market. These four constraints are: logical constraint, conditional constraint, 
succession constraint and timing constraint. 


Contribution 8: An automatic and formal process to use capitalized software bugs in the 
design of test cases suitable to detect similar bugs on a new software development 

Only exhaustive testing can show that a software product is free from bugs. However, 
exhaustive testing of a software product is not practical because variable input values and 
variable sequencing of inputs result in too many possible combinations to test. So it is useful 
to concentrate the test on the areas associated with the greatest risks and priorities. In this 
context, we propose to design fest cases which have a high probability to detect software 
bugs. Therefore, we specify a new format for the “Problem description” attribute of a bug 
capitalized in the problems’ database. This format consists of describing the initial conditions 
and the successive operations that lead to the capitalized bug. Based on this new format, we 
propose an automated process able to design one or more fest cases from each capitalized 
bug. These fest cases are suitable to detect bugs recurrently done by test engineers on specific 
software functionalities. 


Contribution 9: An automatic and formal process to reuse capitalized test cases for one 
project to another 

Reusing capitalized fest cases from one project to another seems to be beneficial in an 
industrial context. In other words, when testing a software functionality that has already been 
implemented in the past on another project, it is judicious to reuse existing fest cases. But 
unfortunately, test cases are not often reused from one project to another. Two potential main 
reasons are: 1) the use of different formats when designing manually fest cases. Sometimes, 
test engineers write the rest cases immediately in a computer language (C language ..….) 
understandable by the test execution platform. Others use a more high level language. 2) the 
lack of an automated process to reuse the rest cases. To overcome this problem, we propose to 
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use one specific format as the standard format to represent a fest case. Based on this new 
format, we develop an automated process able to design one or more fesf cases from each 
capitalized test case. In fact, the designed fest cases focus on test scenarios based on the 
returns of experience from previous projects. 


Contribution 10: Promising results of the experiment of our testing methodology on two 
typical case studies within an automotive electronic supplier 

Through our research project, we propose a new systemic approach to automate efficiently the 
design of test cases for software products. Apart from the computational aspects of software 
testing, the approach takes into account organizational matters (Cf. Contributions 2, 3, 4, 5, 6, 
7, 8 and 9) such as functional simulation, knowledge management, competency management 
and project management. Our testing methodology has been implemented in a computer 
platform and experimented on two typical case studies of Johnson Controls for which 
historical data are available. Consequently, we reduce by 70% of the number of bugs detected 
by the carmakers and by 9% the ones detected by the end-users. Moreover, we reduce by 22% 
the time spent in testing a software product. In fact, we detect the bugs earlier in the software 
development process and closer to their origin. We also propose to deliver to the carmaker 
Jormal quality indicators on the delivered software. AIl these results contribute to an 
improvement of the customer satisfaction and as a direct impact; the number of tenders will 
grow. Unfortunately, estimating the cost of software bugs in an organization 1s a delicate, 
strategic and confidential question and therefore we have not been allowed to communicate 
the numbers on the bugs’ costs savings via the use of our methodology. 


Contribution 11: À patent on our approach to design test cases for software products 

The promising results of the deployment of our testing methodology Within the industrial 
context have motivated the automotive electronic supplier Johnson Controls (who grants this 
PhD) to patent this approach. Presently, the company patent experts are assessing the 
economical profit of patenting our approach. In the meantime, a worldwide Quick Patent” 
(for a preliminary protection of the idea) has been submitted by the company. 


IT. Impact of our testing methodology in the company organization 


Estimating the cost of bugs in a software organization 1s a delicate, strategic and confidential 
question. In 2002, the National Institute of Standards and Technology (NIST) has estimated 
that software bugs cost U.S. economy 59,5 billion dollars annually”. In Johnson Controls, 
there is no model to estimate the cost of software bugs. Unfortunately, these data are 
confidential. However, the number of software bugs detected by the carmakers during 
intermediate deliveries or by the end-users after the Start Of Production (SOP) is estimated 
each month. As the automotive market becomes more and more competing, decreasing the 
development time of outsourced parts and decreasing the number of problems detected later in 
the process becomes of major importance for carmakers and consequently a major quality 
indicator for automotive suppliers. Indeed, the carmakers” process for assigning new projects 
to suppliers is mainly based on feedbacks from previous projects. Through our testing 
methodology (Cf. Table 10.12), we reduce by 70% ((60+80)/2, 60% and 80% respectively on 
the first and second case studies) the number of software bugs detected by the carmakers after 
intermediate deliveries. Making the assumption that the new “minor” bugs that we detect 


* In France, we associate a Quick Patent to an “Enveloppe Soleau” (http://www.inpi.fr/fr/services-et- 
prestations/enveloppe-soleau.html, Consulted on November 2008). 
 http:/www.nist.gov/public_affairs/releases/n02-10.htm (Consulted on November 2008) 
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through our methodology and which were neither detected by Johnson Controls nor by the 
carmaker have been detected by an end-user, we can state that we reduce by 9% ((18+0)/2, 
18% and 0% respectively on the first and second case studies) the number of software bugs 
detected by the end-users once the product is launched on the market. Moreover, we propose 
to deliver to the carmaker quality indicators related to code coverage (already done in the 
industry) but also formal requirements coverage, which may increase its confidence about 
the quality of the software products. Presently, the measurement of requirements coverage is 
informal (Cf. Chapter 2 — Section 6.B.1). In conclusion, across our testing methodology, the 
image of the company (Johnson Controls) in front of its customers (carmakers) will be 
improved and as a direct impact of the customer satisfaction, the number of tenders will 
grow. 


Moreover, the validation test stage accounts for more than 50% of the project time and 
resources (Cf. Chapter 1 — Section 5.C.2). In fact, bugs related to the internal behavior of one 
software module could be detected in unit test stage (earlier in the process). Unfortunately, it 
is not the case and such bugs are detected later in the validation test stage. Of course, 
analyzing the origin of a bug in validation test stage (all the software modules are integrated 
together) 1s more difficult and time consuming than analyzing the bug’s origin in a specific 
software module. Through our testing methodology (Cf. Figure 10.20), we reduce by 22% 
((27+17)/2, 27% and 17% respectively on the first and second case studies) the time spent in 
testing a functionality. While lowering the number of bugs detected by carmakers and end- 
users, we lower the resources required for testing a software product. 


However, we are conscious of the impact of our testing methodology (model and design 
platform) on the current software organization in case of an industrial deployment. 
Indeed, an investment but also personal commitments of all the software players within the 
company are mandatory for the success of such change of practices. In Chapter 9, we develop 
a “process-people-tool” view of our testing methodology. Based on this view, we identify 
three streams of actions necessary for integrating our methodology within the current software 
organization of the company: 


° _Integrate the processes of our methodology (Cf. Figure 9.2) within the global software 
process map of the company (Cf. Figure 2.2). 

e Train the software engineers to the new testing methodology. Test automation has 
broad impacts on an organization such as the skills needed to design and implement 
automated tests, automation tools, and automation environments. The test engineers’ 
practices, roles and competencies change when automation is installed. These impacts 
have negative aspects that must be considered. When introducing a new methodology 
and tool to the testing program, mentors and trainings are very important. Even with 
training, automation skills take time and experience to acquire. The best automation 
tool in the world will not help the test efforts 1f the test team resists using it. The test 
engineers may feel that 1) their manual process works fine, and they don’t want to 
bother with the additional setup work for introducing an automation tool and 2) they 
may lose their know how in designing manually test cases for software products. 
Indeed, test engineers’ technical skills will have to switch from a manual design to a 
high level modeling of the test scenarios and objectives in using in a flexible manner 
our design platform. Nevertheless, based on the literature (Bunse 2007), model-based 
software development approaches are slowly superseding traditional ways of 
developing software products and software engineers’ required skills tend toward 
modeling and automation tool monitoring. 

e  Improve the Man Machine Interfaces of the computer tools that we developed to 
support our testing methodology (Cf. Chapter 8 — Section 3.C and 4). In fact, 
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ergonomic user interfaces play a major role in the practitioners’ use and perception of 
a computer tool. 


III. Research perspectives 


The open perspectives of this research project are listed by topic. 


Perspective 1: Related to the formal language to specify software functional requirements 
The perspectives concerning our formal language to specify software functional requirements 
are: 

e  Perform a broad survey on the carmakers’ specification of the software functional 
requirements. The purpose is to fill out our formal specification language in order to 
be able to specify any carmakers’ software functional requirement. 

e  Develop a list of rules and recommendations to help modelers using efficiently our 
specification language and therefore develop consistent requirements model at the first 
attempt. 

e  Develop more efficient strategies to validate the compliance of a requirements model 
developed using our specification language with the (original) carmaker requirements. 
One solution could be to validate the model by the carmaker itself. 

e  Develop an editor tool to support modelers in designing a requirements model using 
our specification language. For instance, when designing a DT element, designers can 
not consider all the possible conditions on the input signals. In fact, in an industrial 
context, the number of the DT input signals can exceed 10 and the domain length of 
one signal can exceed 100 (for instance, when sampling the “vehicle speed” signal). In 
that case, its remains a very difficult task to identify manually all the possible 
conditions and their corresponding actions. Therefore, an automatic generation of all 
the possible conditions on the input signals of a DT could be judicious. The editor tool 
could perform such functionality. 


Perspective 2: Related to the knowledge management in terms of capitalized bugs and test 
cases 

On the one hand, We propose to reuse capitalized bugs in order to verify the nonexistence of 
recurrent bugs. To do this, we develop a new bug classification model with a detailed 
typology of software problems and a specific format to describe the initial conditions and the 
successive operations that lead to detect a bug. We propose to generate automatically test 
cases that verify the nonexistence of recurrent (capitalized) bugs on each software 
functionality (for instance, front wiper) of a new development. To do this, for each software 
functionality of a product family, a glossary of the functionality’s input signals names on 
previous and new projects are necessary. À family of product is defined by a customer (for 
instance, Renault), à type of product (for instance, a body controller module) and à car 
platform (for instance, Laguna platform). We experiment these proposals on two industrial 
case studies with historical data. However, it could be judicious to experiment our bug 
classification model (software problems typology and description formalism of a bug) and the 
inputs glossary on new software projects. Therefore, we could adjust our proposals in order to 
take practical considerations into account. 

On the other hand, We propose to reuse fest cases from one project to another. To do this, we 
define a new formalism to represent a test case and based on this formalism, we develop an 
automatic process to generate one or more fest cases that focus on operations or successions 
of operations regularly done in a capitalized test case. In fact, we propose to reuse test cases 
when testing a software functionality that we already tested in the past. Therefore, a test cases 
library should be specified in order to capitalize the rest cases by software functionality and 
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family of product. Moreover, for each software functionality of a product family, a glossary of 
the functionality’s input signals names on previous and new projects are necessary. 


Perspective 3: Related to the test case generation algorithm 

Through our experiment, we show that our computational algorithm does not successfully 
reach 100% of functional coverage (the maximum was 90%). Consequently, we were not able 
to detect bugs related to the non-covered functional requirements. To overcome this 
deficiency, we plan to develop a new fest case generation algorithm that focuses on covering 
non-covered zones of a requirements model. In fact and instead of selecting operations via a 
Monte Carlo simulation on the input signals of a model, we propose to synthesize the 
operations that lead in covering a specific item (for instance, a state of an FSM., à condition of 
a DT ...) of the model. In other words, one has to select the item that should be covered and 
the algorithm will propose a list of successive operations to be performed on the model in 
order to cover this item. We already start a global design of this algorithm but unfortunately, 
we had not enough time to implement it in our approach. 


Perspective 4: Related to the strategy for tuning the generation of test cases 


We are conscious of the variability or subjectivity of our current strategy (fry-and-test) to set 
coverage objectives and optimization parameters when generating rest cases. In fact, there are 
a lot of parameters to set. As a consequence, we plan to propose a new strategy to help test 
engineers to parameterize the generation of fest cases. In fact, when testing a software 
product, the main purpose of a test engineer 1s to detect the maximum number of bugs in 
minimum laps of time. Therefore, we have to identify the correlations between the 
optimization algorithm parameters, the functional coverage, the execution time of the 
generated fest cases and the number and type of detected bugs. Based on these correlations, 
we might define rules and recommendations to help test engineers parameterizing the 
generation of test cases. Moreover, we plan to develop parameterization profiles that test 
engineers could adopt according to the test stage objectives. À parameterization profile 
consists of a set of predefined optimization parameters, coverage objectives and time 
constraints. To do this, we plan to perform a Design of Experiments (DOoË) on our approach 
(Cf. Figure Conclusion.1). We set all the functional coverage objectives to 100% with no 
me or cost constraints. We decide to generate test cases for only one “Configuration” of the 
functionality under test (Parameter 6 = 0, Parameter 7 not to be defined). We plan to 
generate one fest case for each combination of the parameters (Parameter 8 = 1). The five 
remaining parameters of the optimization algorithm (Parameter 1, 2, 3, 4 and 5) represent the 
factors of the DoE. Two factors (Parameter 1 and 3) have two levels (0, 1) and three factors 
(Parameter 2, 4 and 5) have n levels (n integer). Based on our experience, we sample the 
domain of these factors into four levels (30, 60, 90 and 120). Consequently, a complete DOoE 
accounts for 256 combinations and a partial one for 16 combinations. We decide to perform 
the partial DoE. After generating one or more fest cases for each combination, we have to 
measure the reached functional coverage. And after executing independently each test case on 
the software module under test, we have to assess the number and type of detected bugs and 
the time spent to execute the rest case. Once all the combinations of the DoE are achieved, the 
experiment results must be analyzed and correlations identified. In fact, we expect that the 
results of the DoE can help test engineers to configure the test platform in a short time 
(around 1 hour) instead of 1 eight-hour day using a fry-and-test strategy (Cf. Chapter 10 — 
Section 8). We start performing the DoE but unfortunately, we had not enough time to 
complete it. 
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Figure Conclusion.1 - A Design of Experiments to identify the correlations between the 


parameters of our approach and the detection of bugs 


Perspective 5: Related to the consistency and reliability of our experiment’s results 

Our approach to design fest cases for software products can be identified to a measurement 
system which has to measure the number of bugs in a software product. As a consequence, we 
have to check the statistical properties of a reliable measurement system: repeatability and 
reproducibility. We start performing this task but unfortunately, we had not enough time to 
complete the experiments. 


Reproducibility: In our testing methodology, the two main activities depend on the 
operator (e.g. human intervention). The first one is the design of the requirements 
model and the second one is the definition of a set of targets and weights for the test 
case generation. Therefore, the reproducibility of our experiment results must be 
verified. In fact, two operators must independently model the same carmaker 
requirements. Rules and recommendations have to be defined in order to help 
operators configure the generation of test cases. Each operator has to generate 
automatically a set of N test cases fulfilling the predefined targets. After executing 
independently each set of N fest cases, one has to assess the ratio of bugs 
simultaneously detected by the two sets of test cases. 

Repeatability: Since our generation of test cases is partly based on a stochastic 
process, the repeatability must be verified. Consequently, we propose to generate two 
or more sets of N test cases from the same requirements model and with the same 
objectives, constraints and optimization parameters. After executing independently 
each set of N test cases, one has to assess the ratio of bugs simultaneously detected by 
two or more sets of fest cases. 
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Perspective 6: Related to the monitoring of our test case design process 
We also plan to monitor the quality of our new testing process. To do so, it seems that within 
the Design for Six Sigma (DFSS) framework, the Define, Measure, Analyze, Design, 
Optimize, and Verify (DMADOV) methodology is the appropriate approach. This will allow us 
to put the proper focus on the up front design of the testing process. Therefore, we need to 
establish the set of measurable, customer-oriented attributes, which can be defined, measured, 
analyzed, optimized and verified (DMADOV) in the software testing process. These attributes 
need to be directly built into the testing process so that it 1s specifically geared to producing 
pre-defined quality limits. This means embedding specific design intent within the software 
testing algorithm to meet specific and understood, customer-facing performance metrics. 
Below, we identify two types of critical-to-customer metrics concerning the software testing 
process. We plan to assess the following metrics on each software project that undergoes 
testing: 
e  Critical-to-Quality (CTO) metrics: 
Y1.The capacity to reduce the number of bugs detected by the carmaker: the ratio 
between the number of bugs detected by carmakers and the total number of bugs 
Y2.The capacity to reduce the number of bugs detected by the end-user: the ratio 
between the number of bugs detected by the end-users and the total number of 
bugs 
e  Critical-to-delivery (CTD) metrics: 
Y3.The number of versions of each software module or product 
Y4.The capacity to deliver software free of bugs since the first delivery: the ratio 
between the number of bugs detected in the first testing phase and the total number 
of bugs 
Since we place a high premium on reducing the number of bugs detected by carmakers and 
end-users (YI and Y2), one solution could be to increase the structural and functional 
coverage. But, experiments reveal that some bugs cannot be detected even 1f our requirements 
model and source code are covered at 100%. This leads to the realization that we need to 
refine our functional coverage model. Typically, we can consider the coverage rate of the 
succession of two transitions in a FSM element. 


IV. General discussions 


In this section, we discuss three major topics related to the deployment and the durability of 
our testing methodology Within an industrial context. 


Since 2003, carmakers, suppliers and other companies from the electronics, semiconductor 
and software industry have been working on the development and introduction of an open, 
standardized software architecture for the automotive industry (AUTOSAR - AUTomotive 
Open System ARchitecture). One of the key features of this consortium is the modularity and 
configurability of automotive software products. This leads to increase the reuse of software 
components from one project to another. As a consequence, reused software components 
would reach a high reliability degree and do not require to be tested unitarily after each reuse. 
Integration and validation test will be of major importance. Nevertheless, the unit test of 
software components will remain necessary since 1) the reused software components 
represent around 50% of the total components of an automotive software product and 2) these 
reused component will evolve continuously (new functionalities and features) and therefore 
need to be tested unitarily. 


Presently, many researches and industrial projects deal with the automatic generation of the 
source code of a software product. The main purposes of these actions are to 1) reduce the 
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software development time and 2) avoid some software problems injected by the software 
engineer when designing and coding the software product. As for the automatic generation of 
test cases, a formal representation of the software specification 1s required. Most of the formal 
specification languages found in the literature attempt to be useful for the code and fest case 
generation. Therefore, it could be useful to explore the automatic generation of source code 
from our functional requirements model of a software product. Considering the following two 
assumptions 1) the requirements model is validated at 100% and 2) the generation of the 
source code 1s reliable at 100%, the generated source code of a software product does not 
need to be tested. Unfortunately, it 1s not the case and a software product needs always to be 
tested (verified and validated). 


Although our testing methodology has been customized to software embedded in cars 
(carmaker requirements formalisms, automotive constraints ...), the use of this approach in 
industries such as aeronautic, railway, medical, telecommunication ... seems beneficial. In 
these industries, software products properties and architectures are similar to the automotive 
industry. However, software requirements formalisms and priorities in testing software 
products could be different. For instance, contrarily to automotive industry, in aeronautic 
industry, constraints on software project planning and budget are less important than software 
quality objectives. This could be explained by the fact that avionics software requires being 
highly reliable, since failures in this kind of products may very likely lead to deathly 
consequences. One more point is the applicability or adaptability of our testing methodology 
to computers applications; for instance, testing software products such as the Microsoft Word 
software. 
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Appendix A: Verification and Validation static tools 


In Johnson Controls, there is a document which defines coding rules and recommendations 
for using the C language“* in the development of embedded software. These rules and 
recommendations are defined and updated by a committee, whose members are appointed by 
the Software Engineering Process Group (SEPG) of the company. The committee includes 
representatives of all Johnson Controls sites on which this document 1s deployed. An excerpt 
of the coding rules and recommendations is illustrated in Table A.I. 


Rule Er 
bee Rule type Rule description 

Rule 1 | General Optimization objectives must be defined before coding. These objectives define 
priorities between optimization ways (memory.…….). 
Do not optimize unless it is planned. 
It has been demonstrated many times that the programmers spend a considerable 
amount of energy to optimize a piece of code that will almost never be used. 
Before starting to optimize always identify the exact nature of the problem. 

Rule 2 | Comments Comments shall be written in US English language. 

Rule 3 | Code layout Each variable must be declared on a separate line. 

Rule 4 | Naming rules Never use names that differ only by uppercase/lowercase. 

Rule 5 | Functions A function must never return à pointer to one if it is a local function. Doing so, 
would rather be a bug than just a rule break. 

Rule 6 | Flow control Give all loops a fixed upper bound. 

Rule 7 | Variables No multiple assignments 
a=b=c=-d; 


Table A.1 — An excerpt of coding rules and recommendations used in Johnson Controls 
(Johnson Controls source) 


The static analysis is performed automatically using a computer tool such as QAC“, the most 
used in automotive industry. The criterion to stop the static analysis of a source code is that 
all QAC errors and warnings are either fixed or justified. À screenshot of the QAC tool is 
illustrated in Figure À.1. 


# Computer language 
% http://www.programmingresearch.com/QAC_ MAIN. html (Consulted on November 2008) 


Quality of the design of test cases for automotive software: design platform and testing process 


295 


Appendix A: Verification and Validation static tools R. AWEDIKIAN 


20 caicc - Microsoft Visual Studio : =ieix| 
Ele Edit View Project Build Debug Tools Window Community Help 

D-3-$44|4 à M|9-0:/%Æ:M| ph» Deby + Wnz2 = | diff.h -AF8x99:s 
14h ll = 20 SSSR me: 


Æ_ 


_ CaiEnvironment.cg@s Sa 19) 
à |? |E 3 CoEnvironment { 
aa *cgiec (1 project) ES | if(stringsAreEqual( getRequestMethod(), "get")) { a 
#È FLE RER | LOGLN("GET method recognized"} Ë 
{n] Coicc.h e LR 
b] c se else if(stringsAreEqual( getRequestMethod(), "post"})) { x 
; LOGLN ("POST method recognized") re 
(n] CoEnvironment.h A 
n] CojUtils.h ë 
FE // should work, but not in eges-1.1.2 or gec-2.95 Ef 
1] FormFile.h /fauto_prr<char> temp = new char[getContentLength()]; 
in] HTMLAttributes.h Char ‘temp = new char{getContentLength()]; 
r] HTMLCIasses.h STDNS cin.read(temp, getContentLength() : 
QAC/++ 
1] HruElements.h 1£(STDNS cin.gcount(} != getContentLength(}} { ET 
] HTMLGeneric.h delete [] temp: M R Gi @ [e] € M LA 
di] HTTPHeaders.h throw STDNS runtime_error ("1/0 error"); 
L] MStreamable.h } 
— fPostData = STDNS string(temp, getContentLength{)): 
EE Source Files 


delete [] temp: 
Ci Coicc.cpp } 

Ci] CoEnvironment,cpp 

G:i Colis. cpp 

> | FormEntry.cpp + 


ÆIsobtion Bo... Feel Property Ma... 


fCookies.reserve (10); 


CgEnvironment.cpp(51,43) 
527 warning QP3050: This is an implic user conversion. CgEnvironment.cpp(54,48) 
526 warning QP4070: This f-else-f chain is not terminated by else. CoEnvironment.cpp(54,8) 


Li 1529 warning QP4263: The class alocates memory in constructor but does nat have an overloaded copy assignment operator. CgEnvironment.cpp(59, 11) 


530 warning QP3000: This is an implicit conversion between signed and unsigned integer types. CoEnwvironment.cpp(60,42) 
31 warning QP3010: This is an implicit conversion from à larger integer to a smaller x tmp CoiEnvironment.cpp(60,42) 


LAcode Defion Wndow [2 1CaN Browser [CZ] Output nie 1 |[$ Error List | 


rend List dfe errors and warnings 


Figure A.1 - Screenshot of the static analysis tool (QAC) 


The dynamic analysis is performed automatically thanks to a commercial tool PolySpace*. 


The criterion to stop the dynamic analysis of a source code is that all Polyspace errors and 
warnings are fixed or justified. À screenshot of the Polyspace tool is illustrated in Figure A.2. 


—, Ds ÆErrors’and warning details 


med (rit Q0 45 pcurm Mme) 
e CÉHCOIEE SZ ET 
roms 


List of errors 
and warnin 


un" 
Demos LC 
= intentions 


& noie nie anttysts 


8 esumpe 


14 


@ Porter_arervmet Software code to be 
analyzed 


Non_rMote _L00p | 


RTE! 


Recurson catier { ) 


Figure A.2 - Sreenshot of the dynamic analysis tool (Polyspace) 


* http:/www.mathworks.com/products/polyspace/index.html (Consulted on November 2008) 
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Appendix B: Test description languages 


As developed in Chapter 2 — Section 5, three techniques of software testing are performed 
before a software delivery to the customer: unit, integration and validation test. In case of unit 
test, the execution of fest cases is always automated using a test execution platform (Cf. 
Appendix C). The language used for describing fest cases for unit test is the C language. 
However, the language used to design test cases for the validation test of a software product 
depends on the validation test execution platform. In case of an automatic execution of the 
test cases, one uses a script language. It is a Johnson Controls property language very similar 
to the well-known Visual Basic‘? language. In case of a manual execution of the test cases, 
test cases are Written in natural language. 


1. Unit test language 


A standard unit test structure provides predefined C functions in order to help test engineers 
writing test cases for the unit test of a software component: 


° Function 1: TSTStartPhase(Title), display Title. 
o The test cases should be broken down in phases to facilitate the test results 
interpretation. 
° Function 2: TSTWaitMs(Delay), wait Delay. 
o This function is essential because the time is only simulated when this function 
1s called. The time is executed at the maximum speed. 
o Delay should be in milliseconds. 
e Function 3: mTSTCheck(Condition), generate an error if Condition is false. 
o Condition is a boolean. 
e Function 4: TSTTerminate(), display Final results of the test 
o List the number of checked points. 
o List the number of bugs. 
o Indicates Test NOK or Test OK if an error has been detected or not. 


An excerpt of test cases designed for the unit test of a software component is 1llustrated in 
Figure B.1. 


*7 Computer language (http:/msdn.microsoft.com/en-us/library/sh9ywfdk(vs.80).aspx, Consulted on November 
2008). 
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// test case 11 
Input 11: 
Input2 12: 
Input3 14: 


TSTWasitMs(S000): 

nTSTCheck(Outputi 
nmTSTCheck(Output2 
nTSTCheck(Output3 


OhoQ 
en mt 


// test case 12 

// OUx0 - essence, 0x10 - diesel 
TYECARE = 0: 

Inputi 0: 

Input2 12: 

Input3 14: 


TSTWaitMs(1000): 

nTSTCheck(Outputlil 
nTSTCheck(Output2 
nmTSTCheck(Output3 


oo 0 
mt 


// test case 13 

// Ox0 - essence, 0x10 - diesel 
TYECARE = 2: 

Inputi 0: 

Input2 12; 

Input3 20; 


Wu un 


TSTWsitMs(S00): 

mTSTCheck(Output1l 
mTSTCheck(Output2 
nTSTCheck(Output3 


mu un 
Wu un 


oOor 
mt 


Figure B.1 —- An excerpt of test cases designed for the unit test of a software component 


(Johnson Controls source) 


2. Integration and Validation test language (when automating the test case execution) 


The test script language developed in Johnson Controls is mainly based on the universal 
Visual Basic language. A set of coding rules and recommendations to be taken into account 
when designing test cases using the test script language has been defined. An excerpt of these 
rules and recommendations is illustrated in Table B.I. 


Rule number 


Description 


Rule 1 


The “include” files must not have the full access path indicated. 
Example (in C ): 

#include "../../h/defs.h" is OK 

#include "defs.h" is OK 

#include <defs.h> is OK 

#include "c:\sources/h/defs.h" is NOK 


Rule 2 


The validation procedures titles must be the same as the title of the SW function (functionality) 
which is testing 


Example: Odometer, Trip meter, Diagnostic, Engine Speed, Vehicle speed, Warnings, etc. 


Rule 3 


Any Test Step having state NOK, must refer to defect reference. 


Rule 4 


Random values in Test Actions and Preconditions are not allowed in any circumstances. Please 
note: Arbitrary values are allowed. Such test types are part of many test procedures. To develop 
more efficient Validation Procedures loop operators shall be used instead linear programming. 


Table B.1 - 


An excerpt of the validation test script coding rules and recommendations 
(Johnson Controls source) 
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The test script is made of a set of sfatements organized in a processes, sub-programs and 
functions structure. The overall structure is described in Figure B.2. 


// Comments 
Scenario “Proba” 
Header Version 02.00 


Include “Models.mvl” 


Declarations 


Include “Library.hvl” 


Const Int State = 0 
ByteArray Values[5] = {0,1,2,3,4} 


(Variables, constants, Int Counter = 0 


Sub-programs and 
Processes.) 


Process Odometer 
// Statements 
End Process 


Main process Process main 


// The main process 


End Process 
Single Empty Line 


Figure B.2 - Overall structure of a test script program (Johnson Controls source) 


nil 


The grammar of the test script language is developed in Figure B.3. 
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. : Integer 
Check <Expression> “The explanation Message” 


StartChecking <Expression> “The explanation 
Message” OnChangeOf <Var> 


Event 


Local variables Integer array 


-_ Checks and errors 


StopChecking <Var Time counter 
Error “The explanation Message” 
Integer 
Wait <Duration> Constants Integer array 
Stops the execution of a process for a a 
given time 
WaitUser “Text of the dialog”, <Varint>, = Integer 
“TxtButton{”, …, ‘TxtButton5” Global variables Event 
Opens a dialog box = Shared by all the X-Car components 7 
Waitings through the Com-center Byte array 


PromptUser "Message" <Var> 
EndOfLine 


Opens a dialog box Usual operators 


R. AWEDIKIAN 


Expressions System of priorities and parenthesis 
SetEvent <VarEvent> 
WaitEvent <VarEvent> [ TimeOut = <Duration> ] Print 
MultipleW ait : 5 
A — Events Display statements |: Display 
- Provide communication channels 
Case <VarEvtN> … between processes De 
[Case TimeOut = <Duration> ] 7e SR 
Enf MultipleW ait Decision structures "Select Case" statement 
sn = = While … Wend 

< > 

rocess <Proc> … End Process Loop structures Do … Loop While 


Calling: 

Starting : StartProcess <Proc> 
Stopping : StopProcess <Proc> 
Waiting : WaitProcess <Proc> 


Declaration: 
Function <FuncName> (<argument list 
declaration>) as Int … End Sub 


Calling: 
<FuncName> (<argument list>) 


Processes 

Piece of code that can be launched 
concurrently from the rest of the script. It 
ends when its last instruction is executed. 


InitTimer ( <VarTimer> ) 


Timers ReadTimer ( <VarTimer>, <Unit> ) 


Declaration: 
Sub <SubName> (<argument list 


Functions declaration>) … End Sub 


Like a Sub-Program but it behaves like an 
expression (i.e. it returns a value) 


Sub-programs 
Named piece of code that can be 
… executed like the language commands 


Calling: 
<SubName> (<argument list>) 


Figure B.3 - Grammar of the test script language 
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As developed in Appendix C, a computer platform (X-Car) has been developed in order to 
execute test cases for the integration and validation test of a software product. This platform 
has a test language interpreter tool which allows to perform initial check for test script 
correctness, run automation fest script, handle data automatically by script and derive output 
for reporting. À screenshot of the script language interpreter tool is ilustrated in Figure B.4. 


Stop 
execution 


? Stat DisgOnCanProcess 


Figure B.4 - Screenshot of the test script interpreter (Johnson Controls sources) 


Another tool 1s the test script sequencer Which allows to manage a list of test scripts in order 
to execute them automatically and consecutively in a specified order. Each script in the list 
has a status and it can be activated or deactivated. Several action can be executed before each 
script (reset the software, reload the software, launch an initialization script). À screenshot of 
the test script sequencer tool is illustrated in Figure B.S. 


2 Valid.bric * - bric 


Ele Exec Setup 2 VE 
(DSE | [4] F2] Application: |C:\Projets NZ \025 0€ VIE 


Root Dir :[C:\Projets\A21025\VAL | où 
E {1 $ (RootDir) a 
5 (9 Parti 
#| VadOdometerl.svl 
&| VaidOdometer2.svl » 
&| ValdT achometerl.svl 
ÿ| VahdT achometer2.svl << 
&]| ValdT achometer3.svl 
5 @ Pat2 
&! ValdFuelConsumption. sv pa] 


Execution finished successfully 
bter24 ! Check error 
® VahidT emperatüre.svl ke Syntax error F| 


Q ValhdT achometerl.svl 


À 
V Ekecution finished successfully 


Card Reader Model (Script 1] - Disactivated 
Card Reader Model (Script 2] - Activated 
Injection Model - Activated 

Initializing Dashboard. Done 


Duretion : Oh 07m 14, 20/Total: 0h07m14,200  Z 


Figure B.5 - Screenshot of the test script sequencer (Johnson Controls source) 
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Appendix C: Test execution platforms 
1. Unit test execution platform 


During the unit test of a software component, the designed fest cases are executed on the 
component automatically via the unit test execution platform. Xn fact, all the dependencies and 
connections between the components are simulated on computer in order to isolate the tested 
component from the project. Test results are analyzed to decide if Component Development 
activities have to be restarted, in case of failed tests. The code coverage is recorded and used 
as criteria to stop the design of fest cases. The abstract model of the unit test execution 
platform is illustrated in Figure C.I. 


Output F(A, Z) 


Software component 


under test Test Cases 
(internal state z) 


Input A 
Figure C.1 - Abstract model of the unit test execution platform (Johnson Controls 
source) 


The unit test uses the inputs and outputs of the software component under test. Test cases 
should know expected output F when input À is applied. The presently produced output has to 
be compared with the expectation. If they do not match, an error should be generated in the 
test report. 


2. Integration and Validation test execution platform 


During the integration and validation tests of a software product, the test execution platform 
could be manual or automatic. For each project, managers (in close cooperation with the 
carmaker) decide to automate or not the execution of the validation test cases. In case of an 
automatic execution, test cases are designed in a script language (Cf. Appendix B). In case of 
a manual execution, test cases are Written in natural language. The manual execution aims to 
perform operations manually on the software product via a set of switches and to check 
visually (by an engineer) the behavior of the output signals (lamps, actuators ...). In the 
following, we develop the automatic test execution platform. 


The Software Validation Plan (SVP) supports the definition of the validation test execution 
platform (Cf. Chapter 2 — Section 5.D.1): the necessary equipments and the common and 
reused validation components. The functional model of a validation test execution platform is 
shown in Figure C.2. 
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Variable power suppl 


Real device on CAN 
BUS) 


Real device on CAN 
BUS 


Simulation and monitoring 
PC 


Direct device inputs 
(Buttons, Analogue 
and Digital signals) 


PC 


CAN(*) 
Network 


( Controller Area Network (or CAN) is the latest communication system within the 
automotive world. At its simplest level, it can be thought of as a means of linking all of 
the electronic systems within a car together to allow them to communicate with each 


other  (htto://www.semiconductors.bosch.de/en/20/can/index.asp Consulted on 
November 2008) 


Figure C.2 - Functional model of a validation test execution platform (Johnson Controls 
source) 


An excerpt of a list of hardware and software tools required for the execution of validation 
test cases 1s 1llustrated in Table C.1 and C2. 


Y/N 


[(HW_T1] | Power supply Constant/variabl 
e/programmable 


Information on the tool configuration 


Specific inputs, outputs and features required 


Work instructions for the tool 


[HW_T2] <Measuring Oscilloscope Y/N 
instrumentation 
> 


Table C.1 - An excerpt from a hardware tool list required for the execution of validation 
test cases 
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[SW_T1] Rio Information for the tool configuration 


de Specific input, output and feature required 


Work instruction for the tool 


Reporter 
1f related with any HW, mention it. 
If standard guideline/work instruction for the 
tool is available, put it as reference document. 
[SW_T2] Test tool 1 R-car Y/N 
Intermat 


Table C.2 — An excerpt from a software tool list required for the execution of validation 
test cases 


Finally, an excerpt of a list of reused components for the execution of validation test cases 1s 
illustrated in Table C.3. 


[SW_V1] Source code Library for PRÉFACE 


programming 


1f related with any HW or SW, mention it. 
power supply 


Path to the original Configuration management 
base 


Work instruction for the tool 


[SW_V2] Test script Test cases for YIN 
<Functionality 
i> 


Table C.3 — An excerpt from a reused components list required for the execution of 
validation test cases 


The aim of the software validation test is to test the functional behavior of a software product 
in its real environment. Therefore, we need to simulate this environment (hardware, other 
electronic devices, network ..…). For that purpose, Johnson Controls has developed two types 
of test execution platform: 


E-Car (Emulated Car — Cf. Figure C.3) is a simulation on computer of the entire electronic 
automotive network with all the electronic devices. This platform simulates also the hardware 
on which the software product under test must perform. It composes network frames on 
specified periods, fills them with the appropriate signals and sends them on a virtual network 
bus. It also simulates pressing of buttons and reaction of sensors in the car. 
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Device 
undertest 


ET 


Device 


undertest 
| 


ardware } 


Network 


Figure C.3 - E-Car environment (Johnson Controls source) 


R-Car (Real Car — Cf. Figure C.4) is a hardware - software interface used when the hardware 
of the software product under test is real physical target. It transforms the parametric signals 
into real electric signals and sends them on the specified channels in the appropriate format. 


Device 
undertest 


Device 
undertest 


1 
L 


7 


Hardware > 
SN 


Network 


Figure C.4 - R-Car environment (Johnson Controls source) 


X-Car is the base framework which allows the running of E-Car or R-Car plus some 
additional programs to support the validation execution (Cf. Figure C.S): 


e__ Network viewer: This tool allows to trace or spy different types of data’s exchange via 
the network. 

e Test language interpreter: It allows to run automation fest scripts and to perform 
initial check for script correctness. It handles data automatically by script and derives 
output for reporting. 

e Bench tool: This tool simulates the inputs and outputs of the device under test. It 
shows data output state, handles data input state and shows data access status. 

e Display simulator: It shows output state by switching between two pictograms, 
simulate pointer indicator on a dashboard and simulate dot matrix display. 
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E-Car or R-Car 


Device 
undertest Communication Center 


Scheduler 


Bench Tool 


Figure C.5 - X-Car framework (Johnson Controls source) 


The core of X-Car is the Communication Center Where all the signals present in the vehicle 
network are stored. Every program that attempt to modify or to check the value of a signal 
will go in there. Another important component is the Scheduler which manages in time the 
platform. 


Integration test may be executed on E-Car; however, validation test may be executed either 
on E-Car or on R-Car. When a bug is detected on E-Car, it must be confirmed on R-Car. In 
fact, E-Car is a simulation on computer while the R-Car is the real physical environment and 
therefore the behavior of the real hardware can differ from the behavior of the simulation 
hardware. 
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Tool name Company Compos Description Input Output pen 
name location trial domain 
Graphiedl-ies A system adapter 
CONFORMIQ Conformiq Test Generator is a solution for | model which Ant . _ a. FES Automotive, 
TEST VERYSOFT | Germany | dynamic model-based test generation and | uses extended D. A YES Aircraft, 
GENERATOR automatic test execution RAS format understood by the Telecommunication 
Don test execution platform 
Test cases are generated 
MaTeLo generates, according to several 2. SE Telecommunication, 
MATELO ALLATEC France optimization algorithms, test cases from a | Usage model in TTCN-3 a NO Automotive, 
usage model TestStand 20 for Railway, Aerospace 
automatic execution 
Pro-Test is a Windows based stand-alone 
tool implementing HTT approach. The goal 
of HTT is to ensure all pairs test case Test cases can be Telecommunication, 
PRO- us " USA coverage With a minimal number of test 2 exported in a variety of YES Railway, Aerospace, 
TEST/PRAXIS TIONS. INC cases . formats including XML, | Defense, PC 
? Praxis is a new service-based HTT solution. Excel, and HTML Software editor 
Praxis offers a custom application of HTT to 
problem based upon specific needs 
PU CENE Reactis automates the generation of test data HN 48e Test Cases are saved in a AHOMONTES 
REACTIS SYSTEMS, USA PAR Stateflow : Sans YES | Telecommunication, 
a from Simulink and Stateflow models special format (“.rst”) : ; 
INC model Aerospace, Medical. 
Rhapsody TestConductor is a UML 
compliant, scenario-based test generation for 
RHAPSODY real-time embedded applications. With 
TESTCONDU L Rhapsody TestConductor, _developers can Automotive, 
CTOR/AUTO LOGIX/TEL. USA test a design against its requirements | DMÉ;diséram. | UML diasram NO Telecommunication, 
MATIC TEST ELOGIC Rhapsody Automatic Test Generator is a Aerospace, Medical, 
EE  —— UML model-based testing solution. ATG Defense 
GENERATOR allows define and 


engineers to test 
individual components for specific purposes 
such as state and transition coverage 


… To be continued 
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mpan mpan ne Fr Application 
Tool name Con es Pany Description Input Output PP mu 
name location trial domain 
T-VEC RAVE: 
T-VEC RAVE solution is a method and T-VEC Test cases can be 
integrated toolset for requirement-based RAVE: T- transformed in test 
T-VEC defect prevention and automated testing. VEC Tabular drivers in any 
RAVE/TESTE System requirements are modeled and RE programming language Automotive, 
R for T-VEC USA with RAVE before design and T-VEC or ue any test YES Pie en 
Simulink/Statefl coding astéps execution t00 — erospace, Medical, 
son T-VEC Tester analyzes Simulink and | A T-VEC Tester: Client-Server, Web 
ow en Simulink and : ee 
Lie Stateflow models for errors, and generates Stateflow Test scripts or drivers are 
comprehensive test cases for verifying the are automatically generated 


models and their implementations 


for executing tests in 
Matlab simulator 


Table D.1 —- Commercial test case design tools (Survey done in 2006) 
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Appendix E: A second-level typology of software problems 


First-level 


Second-level 


Second-level : description 


Requirements incorrect 


The requirement or a part of it is incorrect 


Requirements logic 


The reguirement is illogical or unreasonable 


Requirements completeness 


The requirement as specified is either ambiguous, incomplete, or overly specified 


.… To be continued 


i i : ar. Specification bugs having to do with verifying that the requirement was correctl 
Specification Requirements verifiability jo in : ne ns ; jying 4 4 
update AE à 

: : Bugs in the presentation or documentation of requirements. The requirements are 
Requirements presentation : | | 
presumed to be correct, but the form in which they are presented is not. 
; Requirements, whether or not correct, have been changed between the time 
Requirements changes : à 
programming started and testing ended 
Design correctness Having to do with the correctness of the design 
Design completeness, Feature Having to do with the completeness with which features are designed 
Design completeness, Requirement |Having to do with the completeness of requirements within features 
Processing requirements or feature depends on a combination of input values. À 
Domains domain bug exists if the wrong processing is executed for the selected input-value 
Design update combination 


User messages and diagnostics 


User prompt or printout or other form of communication is incorrect. Processing 
is assumed to be correct: e.g., a false warning, wrong message 


Exception conditions mishandled 


Exception conditions such as failure modes, which require special handling, are 
not correctly handled or the wrong exception-handling mechanisms are used 


Diagnostic conditions mishandled 


Diagnostic conditions, which require special handling, are not correctly handled 
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or the wrong diagnostic-handling mechanisms are used 


Software architecture 


Architectural problems 


Bugs related to the throughput-delay behavior of software under the assumption 


Performance 
: that all other aspects are correct 
Design update / ne _. ; nel j 
- t t titioned, overlay to wrong area, 
Rd emory or virtual memory is incorrectly partition y £ 
overlay or partition conflicts 
Environment Wrong operating system version, or other host environment problem 
Bugs in the interface to third-party software or other software developed 
External and other third-party externally. Due to a misunderstanding or wrong interpretation of the features and 
software operation of the third-party software; or due to problems in the third-party 
software which the vendor does not correct 
oo ] initi initializati ta: e.g., T 
Da RDiton ctuctie, Bugs in the definition, structure and initializa ion of data: e.g., Type, 
: Dimension, Initial values, default values, Duplication, Scope (local, global), 
declaration | : 
Static/dynamic resources 
; Having to do with access and manipulation of data objects that are presumed to 
Data access and handling 8 (ie f ot P 
be correctly defined: e.g. Type, Dimension, Duplication, Resources, Access 
x Bugs specifically related to the control flow of the program or the order and 
Control flow and sequencing 8 specifi A . fl Pier & 
: extent to which things are done, as distinct from what is done 
Implementation 
update Processing Bugs related to processing under the assumption that the control flow is correct 


.… To be continued 


Coding and typographical 


Bugs which can be clearly attributed to simple coding and typographical bugs. 1f 
a programmer believed that the correct variable was ‘'ABCD""' instead of 
"ABCE"" but she/he changed D to E because of a typewriting bug, then it 
belongs to this correction type 


Standards violation 


Bugs having to do with violating or misunderstanding the applicable 
programming standards and conventions (MISRA, Johnson Controls rules ….). 
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Implementation The software is assumed to work properly 
update À Bugs in the documentation associated with the code or the content of comments 
Documentation s : 
contained in the code 
Bugs related to the interfaces between communicating components with the 
program under test. The components are assumed to have passed their component 
level tests. In this context, direct or indirect transfer of data or control information 
Internal interfaces via a memory object such as tables, dynamically allocated resources, or files, 
Integration constitute an internal interface (e. g. Component invocation, Interface parameter, 
update Component invocation return, Invocation in wrong place, Duplicate invocation, 
_—. 
Having to do with external interfaces, such as 1/0 devices and/or drivers, or other 
External interfaces and timing software not operating under the same control structure (e. g. Interrupts, Devices 
and drivers, 1/0 timing) 
Manufacturin ; 
8 Manufacturing bugs Bugs related to the manufacturing process 
update 
Test design bugs Bugs in the design of tests 
Test execution bugs Bugs in the execution of tests 
Test Case update , ,  . ee , 
Test documentation Documentation of test case or verification criteria is incorrect or misleading 
Test case completeness Cases required to achieve specified coverage criteria missing 
Update none None None of the proposed types of problem is applicable 


Table E.1 — A second-level typology of software problems 
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